Professional Documents
Culture Documents
HP Vertica 7.1.x AdministratorsGuide
HP Vertica 7.1.x AdministratorsGuide
Legal Notices
Warranty
The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be
construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
The information contained herein is subject to change without notice.
Copyright Notice
Copyright 2006 - 2015 Hewlett-Packard Development Company, L.P.
Trademark Notices
Adobe is a trademark of Adobe Systems Incorporated.
Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation.
UNIX is a registered trademark of The Open Group.
Page 2 of 1217
Contents
Contents
Administration Overview
65
Managing Licenses
66
Copying Enterprise, Evaluation, Flex Zone, and SQL on Hadoop License Files
66
66
67
67
68
70
70
71
71
72
72
72
73
73
73
74
75
75
75
75
76
76
Page 3 of 1217
Administrator's Guide
Contents
77
77
77
78
Targeted Auditing
78
79
79
79
80
If Your HP Vertica Enterprise Edition Database Size Exceeds Your Licensed Limits
80
81
81
83
Configuration Procedure
84
IMPORTANT NOTES
84
85
86
86
Notes
86
86
87
87
88
88
88
88
89
Page 4 of 1217
Administrator's Guide
Contents
89
90
90
90
91
92
92
92
94
94
Notes
95
96
98
99
99
99
100
100
101
102
102
102
104
104
105
Server Locale
105
vsql Client
105
Page 5 of 1217
Administrator's Guide
Contents
ODBC Clients
106
106
Usage Considerations
106
109
Configuration Parameters
111
111
113
113
113
115
General Parameters
116
120
Projection Parameters
122
123
Monitoring Parameters
124
Profiling Parameters
126
Security Parameters
126
128
Internationalization Parameters
129
130
131
132
134
135
135
135
137
Page 6 of 1217
Administrator's Guide
Contents
Creating Schemas
138
138
139
141
Tables in Schemas
142
142
142
143
144
144
Implementing Views
147
Creating Views
147
Using Views
148
149
151
What Is a Design?
151
152
154
154
155
156
157
158
160
160
161
Page 7 of 1217
Administrator's Guide
Contents
DC_DESIGN_PROJECTION_CANDIDATES
161
DC_DESIGN_QUERY_PROJECTION_CANDIDATES
161
162
166
Design Name
167
Design Types
167
Comprehensive Design
167
Incremental Design
168
Optimization Objectives
168
168
Design Queries
169
Query Repository
K-Safety for Design
K-Safety Requirements
Replicated and Unsegmented Projections
169
169
170
170
Replicated Projections
171
Unsegmented Projections
171
Statistics Analysis
172
Building a Design
172
Resetting a Design
173
Deploying a Design
174
174
175
175
177
177
179
Page 8 of 1217
Administrator's Guide
Contents
183
185
186
186
188
188
DBDUSER Privileges
189
189
194
194
195
Design Requirements
195
196
196
Monitoring K-Safety
197
Loss of K-Safety
197
197
198
198
Segmenting Projections
199
199
200
201
202
202
Designing Superprojections
203
Minimizing StorageRequirements
203
Page 9 of 1217
Administrator's Guide
Contents
203
204
206
206
206
207
208
208
209
209
Sort Foreign Key Columns First, From Low to High Distinct Cardinality
209
210
212
212
213
214
215
215
216
Implementing Security
218
Client Authentication
218
Connection Encryption
218
Client Authorization
219
Client Authentication
220
220
222
222
Page 10 of 1217
Administrator's Guide
Contents
223
224
226
228
230
231
232
233
235
235
Password Authentication
237
237
237
Profiles
238
238
239
239
Expiring a Password
240
Account Locking
Unlock a Locked Account
Password Guidelines
Configuring LDAPAuthentication
What You Need to Know to Configure LDAPAuthentication
240
240
241
242
242
243
LDAPAuthentication Definitions
243
DBADMINAuthentication Access
244
LDAPParameters
245
Page 11 of 1217
Administrator's Guide
Contents
248
LDAPAnonymous Binding
249
249
251
251
253
Configuring MultipleLDAPServers
254
254
Kerberos Requirements
255
256
256
256
Principals
257
Keytab Files
257
258
259
260
261
261
Restartthe Database
262
262
262
Configure ODBC and vsql Clients on Linux, HP-UX, AIX, MAC OSX, and
Solaris
263
265
266
266
Page 12 of 1217
Administrator's Guide
Contents
267
267
268
271
271
271
271
272
272
272
273
273
273
276
277
278
278
278
279
279
Configuring TLSAuthentication
280
282
282
283
284
284
284
Page 13 of 1217
Administrator's Guide
Contents
285
286
Implementing SSL
288
Certificate Authority
288
289
SSL Prerequisites
289
289
290
291
291
292
293
295
295
JDBC Certificates
JDBC Certificates Summary
Generating Certificates and Keys for MC
296
297
298
Signed Certificates
298
Self-Signed Certificates
299
300
300
300
Configuring SSL
301
302
303
303
304
Page 14 of 1217
Administrator's Guide
Contents
304
304
304
306
307
308
DBADMIN User
308
Object Owner
308
PUBLIC User
309
309
Notes
309
Example
310
310
311
311
313
313
MC User Types
313
314
Default MC Users
314
Creating an MC User
314
Prerequisites
315
316
317
317
Managing MC Users
Who Manages Users
318
318
Page 15 of 1217
Administrator's Guide
Contents
319
319
320
321
321
321
Schemas
321
Tables
322
Views
324
Projections
324
External Procedures
325
Libraries
325
User-Defined Functions
325
Sequences
326
Resource Pools
327
Users/Profiles/Roles
328
Object Visibility
328
I/O Operations
330
Comments
331
Transactions
331
Sessions
332
Tuning Operations
332
333
Database Privileges
333
Schema Privileges
334
335
336
Page 16 of 1217
Administrator's Guide
Contents
Projection Privileges
337
337
337
338
Dropping Projections
338
View Privileges
338
Sequence Privileges
339
340
340
Library Privileges
341
341
342
343
Metadata Privileges
344
I/O Privileges
345
Comment Privileges
346
Transaction Privileges
346
Session Privileges
347
Tuning Privileges
347
347
347
348
348
349
350
351
Modifying Privileges
354
Page 17 of 1217
Administrator's Guide
Contents
354
Notes
354
Example
354
356
357
358
358
360
362
363
364
364
365
365
366
Role Hierarchies
366
366
367
368
DBADMIN Role
View a List of Database Superusers
368
369
DBDUSER Role
369
PSEUDOSUPERUSER Role
370
PUBLIC Role
371
Example
Default Roles for Database Users
Notes
371
371
372
Page 18 of 1217
Administrator's Guide
Contents
372
Role Hierarchy
373
Example
373
375
375
376
Example
376
377
377
Example
Revoking Access From Database Roles
Example
Granting Administrative Access to a Role
Example
Revoking Administrative Access From a Role
377
379
379
380
380
381
Example
381
Enabling Roles
381
Disabling Roles
382
382
383
383
383
Users
384
Roles
385
Grants
385
385
387
Page 19 of 1217
Administrator's Guide
Contents
MC Privilege Groups
387
387
MC Configuration Privileges
388
389
389
391
392
393
394
394
395
395
396
MC Database Privileges
396
397
398
399
400
400
401
401
401
402
402
Prerequisites
402
402
403
Page 20 of 1217
Administrator's Guide
Contents
404
404
407
407
408
409
409
409
MC Mapping Matrix
410
413
413
414
Between Dialogs
414
415
Enter [Return]
415
OK - Cancel - Help
415
Menu Dialogs
416
List Dialogs
416
Form Dialogs
416
Help Buttons
417
417
418
419
In a Menu Dialog
420
In a Dialog Box
420
Scrolling
421
Password Authentication
421
Page 21 of 1217
Administrator's Guide
Contents
421
422
425
425
426
427
427
427
428
Stopping a Database
429
Error
429
Description
429
Resolution
429
Controlling Sessions
432
Notes
433
433
435
Creating a Database
435
Dropping a Database
437
Notes
437
Viewing a Database
438
438
439
440
442
442
442
Page 22 of 1217
Administrator's Guide
Contents
443
444
445
Managing Clusters
445
446
446
446
447
Syntax
447
Parameters
448
Tools
448
458
Connecting to MC
458
459
460
461
461
Notes
462
463
463
464
464
466
468
Managing MC Settings
Modifying Database-Specific Settings
Changing MC or Agent Ports
468
469
470
Page 23 of 1217
Administrator's Guide
Contents
470
470
470
471
471
Backing Up MC
472
472
472
473
475
475
475
476
476
477
477
477
Starting over
478
478
478
481
481
481
481
482
482
483
Page 24 of 1217
Administrator's Guide
Contents
483
484
485
Viewing Results
485
Privileges
485
486
486
486
487
487
Reviewing Errors
488
Managing Tables
Creating Base Tables
491
491
491
492
493
495
495
496
Simple Example
496
496
498
498
498
499
500
501
Page 25 of 1217
Administrator's Guide
Contents
501
502
502
503
503
504
504
504
505
505
505
505
505
506
508
508
508
508
509
509
509
510
510
511
511
512
512
Page 26 of 1217
Administrator's Guide
Contents
512
513
513
513
514
515
516
Examples
517
518
520
520
Examples
520
521
Restrictions
521
521
Examples
522
523
524
Notes
524
Example
524
526
527
528
528
529
530
530
Page 27 of 1217
Administrator's Guide
Contents
531
532
532
Creating Sequences
533
Altering Sequences
535
Examples
535
Distributed Sequences
536
Loading Sequences
546
546
547
Dropping Sequences
Example
Synchronizing Table Data with MERGE
547
548
548
548
551
552
Dropping Tables
552
Truncating Tables
552
554
554
Creating a TextIndex
555
556
558
About TxtIndex.Stemmer
559
Working withProjections
560
Types of Projections
Live Aggregate Projections
560
561
Page 28 of 1217
Administrator's Guide
Contents
Top-KProjections
561
Projection Basics
561
562
562
562
563
563
564
565
567
567
Supported Aggregations
569
Top-K Projections
569
571
571
573
574
575
576
576
578
579
580
System Table Fields for Live Aggregate Projections and Projections with Expressions
581
About Constraints
585
Adding Constraints
586
586
Page 29 of 1217
Administrator's Guide
Contents
587
587
588
589
590
Enforcing Constraints
590
591
592
Examples
592
Unique Constraints
593
595
Dropping Constraints
Notes
Enforcing Primary Key and Foreign Key Constraints
596
596
598
598
598
598
599
604
607
608
609
Defining Partitions
610
612
612
612
613
Page 30 of 1217
Administrator's Guide
Contents
Dropping Partitions
Examples
613
614
Swapping Partitions
615
616
616
Partition Pruning
616
Managing Partitions
617
Notes
Partitioning, Repartitioning, and Reorganizing Tables
618
618
619
Monitoring Reorganization
619
Auto Partitioning
620
Examples
620
Eliminating Partitions
623
624
624
Examples
625
Moving Partitions
626
Archiving Steps
626
626
627
628
628
628
631
632
633
Page 31 of 1217
Administrator's Guide
Contents
633
633
634
634
635
635
636
636
636
636
637
637
638
639
639
639
639
640
640
Transformation Example
641
641
642
643
643
643
644
644
Page 32 of 1217
Administrator's Guide
Contents
644
645
645
645
TRICKLELoading (Incremental)
646
646
647
647
647
647
648
648
650
650
651
652
652
653
653
654
654
655
656
657
657
657
659
Page 33 of 1217
Administrator's Guide
Contents
659
Examples
660
660
661
661
Hexadecimal Data
663
Octal Data
663
BitString Data
664
Examples
664
665
665
665
666
666
667
668
669
669
670
Syntax
670
Usage
671
672
673
674
676
676
677
Page 34 of 1217
Administrator's Guide
Contents
677
677
677
678
678
678
679
680
681
681
681
682
682
683
685
686
687
687
687
689
690
691
692
692
693
693
694
Page 35 of 1217
Administrator's Guide
Contents
695
695
696
697
697
697
Exporting Data
698
699
699
Copying Data
701
701
Examples
702
703
703
704
705
706
706
Function Summary
706
707
Projection Considerations
707
708
708
Exporting Tables
709
Function Syntax
709
709
710
Page 36 of 1217
Administrator's Guide
Contents
710
711
Function Syntax
711
711
713
714
715
716
717
717
718
718
719
720
721
721
722
722
Disabling Purge
723
723
725
725
726
IPVSOverview
726
726
727
729
Page 37 of 1217
Administrator's Guide
Contents
729
730
731
731
732
734
Notes
735
735
736
739
739
739
739
740
741
741
742
744
744
746
747
Issue
747
Resolution
747
748
748
750
750
Page 38 of 1217
Administrator's Guide
Contents
751
751
Node Types
751
752
753
753
753
754
754
754
754
754
754
755
757
759
759
761
761
761
761
762
762
762
Large Cluster
763
763
764
Page 39 of 1217
Administrator's Guide
Contents
764
765
765
766
768
768
770
770
771
771
771
772
772
773
774
774
774
774
Elastic Cluster
774
775
776
776
776
777
777
778
778
Page 40 of 1217
Administrator's Guide
Contents
779
779
780
780
Adding Nodes
781
782
782
783
Examples:
784
784
784
785
Removing Nodes
787
787
788
Prerequisites
788
788
Remove Unused Hosts From the Database Using the Administration Tools
788
789
Prerequisites
789
790
Replacing Nodes
791
Prerequisites
791
792
792
793
794
Page 41 of 1217
Administrator's Guide
Contents
795
Replace the Original Host with a New Host Using the Administration Tools
Using the Management Console to Replace Nodes
795
796
797
797
798
Permissions
798
798
799
799
800
801
802
802
802
804
804
805
807
Rebuilding a Table
Notes
807
807
809
810
Moveout
ROS Containers
Mergeout
Mergeout of Deletion Markers
811
811
811
812
Page 42 of 1217
Administrator's Guide
Contents
813
813
814
Loading Data
815
816
816
Managing Workloads
817
Statements
817
System Tables
818
819
819
821
821
821
822
822
822
823
Parameter Considerations
823
825
Examples
825
User Profiles
829
Example
830
831
832
832
833
Page 43 of 1217
Administrator's Guide
Contents
833
835
835
Using CHANGE_RUNTIME_PRIORITY
835
836
837
837
837
Usage Considerations
843
844
845
845
846
847
848
849
851
852
852
853
854
856
856
856
856
857
857
Page 44 of 1217
Administrator's Guide
Contents
Real-Time Profiling
Managing System Resource Usage
Managing Sessions
858
859
860
Viewing Sessions
860
861
Controlling Sessions
861
863
864
864
864
865
865
865
866
867
867
867
868
869
870
870
871
872
872
872
874
875
Page 45 of 1217
Administrator's Guide
Contents
875
876
878
878
879
879
880
881
882
882
883
884
885
886
886
887
887
888
888
889
889
889
889
890
893
893
893
Page 46 of 1217
Administrator's Guide
Contents
Types of Backups
894
Full Backups
894
Object-Level Backups
895
895
Related Tasks
895
896
896
896
897
897
898
898
898
899
See Also
900
Related Terms
900
Related Tasks
900
900
900
902
902
903
903
904
904
904
905
Page 47 of 1217
Administrator's Guide
Contents
905
905
906
906
See Also
906
Related Tasks
906
906
907
See Also
908
Related Tasks
908
908
910
See Also
910
Related Tasks
910
910
910
911
911
911
911
912
913
913
913
914
See Also
914
Related Tasks
915
Page 48 of 1217
Administrator's Guide
Contents
915
915
916
Object-Level Backups
917
917
917
918
See Also
918
Related Terms
919
Related Tasks
919
919
Directory Tree
919
920
Related Tasks
921
922
922
923
923
Naming Conventions
923
924
924
924
925
926
926
927
927
Page 49 of 1217
Administrator's Guide
Contents
Backup Epochs
928
928
See Also
928
Related Tasks
928
928
929
929
Related Terms
930
931
Viewing Backups
931
931
Monitor database_snapshots
932
Query database_backups
932
Related Terms
933
Related Tasks
933
933
934
Restoring an Archive
935
935
935
See Also
936
Related Terms
936
936
Backup Locations
936
937
937
937
Page 50 of 1217
Administrator's Guide
Contents
938
938
See Also
939
Related Terms
939
Related Tasks
939
939
939
939
941
942
See Also
942
Related Tasks
943
Removing Backups
Deleting Backup Directories
Copying the Database to Another Cluster
943
943
944
945
946
946
947
Related Terms
948
Related Tasks
948
949
949
Syntax
949
Parameters
950
See Also
951
Related Tasks
951
Page 51 of 1217
Administrator's Guide
Contents
951
951
954
955
[Mapping]
957
Related Terms
958
Related Tasks
958
958
959
961
961
Recovery Scenarios
962
Notes
963
964
964
965
965
968
Monitoring Recovery
968
968
969
969
969
Exporting a Catalog
970
970
971
Monitoring HP Vertica
973
Page 52 of 1217
Administrator's Guide
Contents
973
973
973
974
974
975
975
977
978
979
979
980
980
981
Targeted Auditing
981
982
982
984
984
Monitoring Parameters
984
Monitoring Events
987
987
987
Event Data
991
995
995
996
Page 53 of 1217
Administrator's Guide
Contents
996
997
998
999
1000
1000
1000
1001
1002
1002
Vertica.log
1002
SNMP
1002
Syslog
1003
1004
1004
1005
1005
Examples
1006
1008
Data Collector
1008
1008
DC Tables
1009
1009
1010
1011
1012
1013
Page 54 of 1217
Administrator's Guide
Contents
1013
1014
1015
1015
1016
1017
1017
1017
Examples
Monitoring Recovery
1018
1021
1022
1022
1022
1023
1025
1025
1026
Tasks
1026
Recent Databases
1027
1027
1028
Quick Stats
1028
Status Summary
1030
System Health
1031
Query Synopsis
1032
1034
1034
Page 55 of 1217
Administrator's Guide
Contents
1034
1035
1035
1035
1036
1036
1037
1037
1038
1038
1039
1039
1039
1041
1041
1041
1041
Chart Results
1042
1042
1043
1043
1043
1044
1044
Example
1044
1044
1046
Page 56 of 1217
Administrator's Guide
Contents
1046
1047
1049
1050
1052
1052
1055
Message Severity
1056
1056
1057
1057
1057
1058
1059
1059
1063
1064
1064
1064
Analyzing Workloads
1065
1065
1065
1065
1066
1067
1067
1067
Page 57 of 1217
Administrator's Guide
Contents
1068
Examples
1068
1069
1070
1070
1071
1079
1080
1081
1081
Using ANALYZE_STATISTICS
1081
Using ANALYZE_HISTOGRAM
1082
Using ANALYZE_EXTERNAL_ROW_COUNT
1083
1084
1085
1085
1090
Example
1091
1092
1092
1093
Save Statistics
1094
1096
1096
1096
Requirements
1097
Further Information
1097
Page 58 of 1217
Administrator's Guide
Contents
Running scrutinize
Command Options
Limiting the Scope of scrutinize Data Collection
1097
1098
1104
1105
1106
1106
1107
scrutinize Security
1108
Troubleshooting scrutinize
1109
1110
1111
Exporting a Catalog
1112
1112
Syntax
1113
Parameters
1113
Example
1113
1114
1115
1116
1117
1118
1120
1121
1122
1122
1123
1124
Page 59 of 1217
Administrator's Guide
Contents
1124
1126
1127
1128
1130
1130
1131
Inequality joins
1133
1133
1134
1135
1135
GROUPBYHASHQueryPlan Example
1136
1137
1137
1139
1139
1140
1141
1143
1144
1146
1149
1150
1150
1151
1152
Page 60 of 1217
Administrator's Guide
Contents
1152
1153
1153
1154
1154
1156
1159
1159
1160
1160
Projection Metadata
1160
1161
Profile Metrics
1162
Execution Events
1163
Optimizer Events
1165
1165
1165
1166
1166
1167
1167
1168
1168
1168
1169
1169
1170
Page 61 of 1217
Administrator's Guide
Contents
Label syntax
1170
1172
1172
1173
1174
1176
1177
1178
1178
About Locales
1179
1179
Locales
1179
Notes
1180
1180
1181
Locale Specification
1183
Long Form
1183
Syntax
1183
Parameters
1183
1187
Usage Considerations
1189
Examples
1190
Short Form
1191
1191
1191
Supported Locales
1192
1204
Page 62 of 1217
Administrator's Guide
Contents
1207
1207
File Signature
1207
Column Definitions
1207
Row Data
1211
Example
1212
1217
Page 63 of 1217
Administrator's Guide
Contents
Page 64 of 1217
Administrator's Guide
Administration Overview
Administration Overview
This document describes the functions performed by an HP Vertica database administrator (DBA).
Perform these tasks using only the dedicated database administrator account that was created
when you installed HP Vertica. The examples in this documentation set assume that the
administrative account name is dbadmin.
l
To perform certain cluster configuration and administration tasks, the DBA (users of the
administrative account) must be able to supply the root password for those hosts. If this
requirement conflicts with your organization's security policies, these functions must be
performed by your IT staff.
If you perform administrative functions using a different account from the account provided
during installation, HP Vertica encounters file ownership problems.
If you share the administrative account password, make sure that only one user runs the
Administration Tools at any time. Otherwise, automatic configuration propagation does not
work correctly.
The Administration Tools require that the calling user's shell be /bin/bash. Other shells give
unexpected results and are not supported.
Page 65 of 1217
Administrator's Guide
Managing Licenses
Managing Licenses
You must license HP Vertica in order to use it. Hewlett-Packard supplies your license information
to you in the form of one or more license files, which encode the terms of your license. Several
licenses are available:
l
vlicense_565_bytes.dat, for data stored in a Hadoop environment with HPVertica for SQL on
Hadoop.
To prevent introducing special characters (such as line endings or file terminators) into the license
key file, do not open the file in an editor or e-mail client. Though special characters are not always
visible in an editor, their presence invalidates the license.
Page 66 of 1217
Administrator's Guide
Managing Licenses
Raw data size based (valid to store up to some amount of raw data)
Raw data size based and a limit of 3 nodes (HP Vertica Community Edition)
Node-based and an unlimited number of CPUs and users (one node is a server acting as a single
computer system, whether physical or virtual)
Your license key has your licensing bases encoded into it. If you are unsure of your current license,
you can view your license information from within HP Vertica.
HPVertica for SQLon Hadoop is a separate product with its own license. This documentation
covers both products.
3 node limit
Community Edition licenses cannot be installed co-located in a Hadoop infrastructure and used to
query data stored in Hadoop formats.
HP Vertica Enterprise Edition. You can purchase the Enterprise Edition license. The Enterprise
Edition license entitles customers to:
Page 67 of 1217
Administrator's Guide
Managing Licenses
No node limit
Query data stored in HDFS using the HCatalog Connector or HDFSConnector, and back up HP
Vertica data to HDFS
Enterprise Edition licenses cannot be installed co-located in a Hadoop infrastructure and used to
query data stored in Hadoop formats, and exclude the ability to read ORC Files directly.
Flex Zone. Flex Zone is a license for the flex tables technology, available in version 7.0.
Customers can separately purchase and apply a Flex Zone license to their installation. The Flex
Zone license entitles customers to the licensed amount of Flex table data and removes the 3 node
restriction imposed by the Community Edition.
Customers whose primary goal is to work with flex tables can purchase a Flex Zone license. When
they purchase Flex Zone, customers receive a complimentary Enterprise License, which entitles
them to one terabyte of columnar data and imposes no node limit.
Note: Customers who purchase a Flex Zone license must apply two licenses: their Enterprise
Edition license and their Flex Zone license.
Allowances
Community
Edition
Enterprise
Edition
Flex
Zone
Node Limit
3 nodes
Unlimited
Unlimited
Unlimited
Columnar
Data
1 terabyte
Per license
Per license
Flex Data
1 terabyte
terabyte
1 terabyte
Per license
Per
license
Page 68 of 1217
Administrator's Guide
Managing Licenses
Customers can purchase this term-based SQLon Hadoop license per the number of nodes they
plan to use in their Hadoop environment. The license then audits the number of nodes being used
for compliance.
Users have access to a subset of the Enterprise Edition features with this license.
Features
Enterprise Edition
SQL on Hadoop
HP Vertica Distributed R
Yes
No
HPVertica Pulse
Yes
No
HPVertica Place
Yes
No
Key Value
Yes
Yes
Yes
No
Yes
Yes
Text Search
Yes
No
Workload Management
Yes
Yes
Advanced Workload
Yes
No
Database Designer
Yes
Yes
Management Console
Yes
Yes
Yes
No
Flex Zone
Yes
Yes
External Tables
Yes
Yes
File Formats
Formats
Formats
Yes
Yes
No
Yes
Management (Cascading
Resource Pools)
Page 69 of 1217
Administrator's Guide
Managing Licenses
MPPSQL Engine
Yes
Yes
Hadoop Distribution
Hortonworks, Cloudera,
Compatibility
MapR
Java User-Defined
Yes
Yes
Yes
No
R User-Defined Extensions
Yes
No
Extensions
Page 70 of 1217
Administrator's Guide
Managing Licenses
Note: If you installed Management Console, the MC administrator can point to the
location of the license key during Management Console configuration.
Page 71 of 1217
Administrator's Guide
Managing Licenses
Note: As soon as you renew or upgrade your license key from either your Administration
Host or Management Console, HP Vertica applies the license update. No further warnings
appear.
Page 72 of 1217
Administrator's Guide
Managing Licenses
Or, use the LICENSES table described in the SQL Reference Manual to view information about all
your installed licenses. This table displays your license types, the dates for which your licenses are
valid, and the size and node limits your licenses impose. In the example below, the licenses table
Page 73 of 1217
Administrator's Guide
Managing Licenses
displays the Community Edition license and the default license that controls HP Vertica's flex data
capacity.
=> SELECT * FROM licenses; \x
-[RECORD 1 ]--------+---------------------------------------license_id
| 45035996273704986
name
| vertica
licensee
| Vertica Community Edition
start_date
| 2011-11-22
end_date
| Perpetual
size
| 1TB
is_community_edition | t
node_restriction
| 3
-[RECORD 2 ]--------+---------------------------------------license_id
| 45035996274085644
name
| com.vertica.flextable
licensee
| Vertica Community Edition, Flextable
start_date
| 2013-10-29
end_date
| Perpetual
size
| 1TB
is_community_edition | t
node_restriction
|
You can also view the LICENSES table inManagement Console. On your database's Overview
page in Management Console, click the License tab. The License page displays information about
your installed licenses.
Page 74 of 1217
Administrator's Guide
Managing Licenses
Multiple projections (underlying physical copies) of data from a logical database entity (table).
Data appearing in multiple projections of the same table is counted only once.
Page 75 of 1217
Administrator's Guide
Managing Licenses
Data that has been deleted, but which remains in the database. To understand more about
deleting and purging data, see Purging Deleted Data.
Data stored in system and work tables such as monitoring tables, Data Collector tables, and
Database Designer tables.
Delimiter characters.
Strings and binary types (CHAR, VARCHAR, BINARY, VARBINARY) are counted as their
actual size in bytes using UTF-8 encoding.
Numeric data types are counted as if they had been printed. Each digit counts as a byte, as does
any decimal point, sign, or scientific notation. For example, -123.456 counts as eight bytes (six
digits plus the decimal point and minus sign).
Date/time data types are counted as if they had been converted to text, including any hyphens or
other separators. For example, a timestamp column containing the value for noon on July 4th,
2011 would be 19 bytes. As text, vsql would print the value as 2011-07-04 12:00:00, which is 19
characters, including the space between the date and the time.
Page 76 of 1217
Administrator's Guide
Managing Licenses
Note: Unlike estimating raw data size using statistical sampling, a full audit performs SQL
queries on the full database contents, including the contents of the WOS.
Page 77 of 1217
Administrator's Guide
Managing Licenses
The size of the database is reported in bytes. The AUDIT function also allows you to control the
accuracy of the estimated database size using additional parameters. See the entry for the AUDIT
function in the SQL Reference Manual for full details. HP Vertica does not count the AUDIT
function results as an official audit. It takes no license compliance actions based on the results.
Note: The results of the AUDIT function do not include flexible table data. Use the AUDIT_
FLEXfunction to monitor data usage for your Flex Tables license .
Targeted Auditing
If audits determine that the columnar table estimates are unexpectedly large, consider schemas,
tables, or partitions are using the most storage. You can use the AUDIT function to perform
targeted audits of schemas, tables, or partitions by supplying the name of the entity whose size you
want to find. For example, to find the size of the online_sales schema in the VMart example
database, run the following command:
Page 78 of 1217
Administrator's Guide
Managing Licenses
You can also change the granularity of an audit to report the size of each entity in a larger entity (for
example, each table in a schema) by using the granularity argument of the AUDIT function. See the
AUDIT function's entry in the SQL Reference Manual.
The needle in the license meter adjusts to reflect the amount used in megabytes.
The Audit button returns the same information as the AUDIT() function in a graphical
representation.
The Details link within the License grid (next to the Audit button) provides historical information
about license usage. This page also shows a progress meter of percent used toward your
license limit.
Page 79 of 1217
Administrator's Guide
Managing Licenses
Delete data from your database to remain under your licensed raw data size allowance. The
warnings disappear after HP Vertica's next audit of the database size shows that it is no longer
close to or over the licensed amount. You can also manually run a database audit (see
Monitoring Database Size for License Compliance for details).
If your database continues to grow after you receive warnings that its size is approaching your
licensed size allowance, HP Vertica displays additional warnings in more parts of the system after
a grace period passes.
Page 80 of 1217
Administrator's Guide
Managing Licenses
where:
l
database must be a running database. If the database is passport protected, you must also
supply the password.
--file csv-file directs output to the specified file. If csv-file already exists, the tool returns
with an error message. If this option is unspecified, output is directed to stdout.
--quiet specifies to run the tool in quiet mode; if unspecified, status messages are sent to
stdout.
Running the license_audit tool is equivalent to invoking the following SQL statements:
select
select
select
select
select
audit('');
audit_flex('');
* from dc_features_used;
* from vcatalog.license_audits;
* from vcatalog.user_audits;
Page 81 of 1217
Administrator's Guide
Managing Licenses
Audit timestamps
The following truncated example shows the raw CSV output that license_audit generates:
FEATURES_USED
features_used,feature,date,sum
features_used,metafunction::get_compliance_status,2014-08-04,1
features_used,metafunction::bootstrap_license,2014-08-04,1
...
LICENSE_AUDITS
license_audits,database_size_bytes,license_size_bytes,usage_percent,audit_start_
timestamp,audit_end_timestamp,confidence_level_percent,error_tolerance_percent,used_
sampling,confidence_interval_lower_bound_bytes,confidence_interval_upper_bound_
bytes,sample_count,cell_count,license_name
license_audits,808117909,536870912000,0.00150523690320551,2014-08-04 23:59:00.02487404,2014-08-04 23:59:00.578419-04,99,5,t,785472097,830763721,10000,174754646,vertica
...
USER_AUDITS
user_audits,size_bytes,user_id,user_name,object_id,object_type,object_schema,object_
name,audit_start_timestamp,audit_end_timestamp,confidence_level_percent,error_tolerance_
percent,used_sampling,confidence_interval_lower_bound_bytes,confidence_interval_upper_
bound_bytes,sample_count,cell_count
user_audits,812489249,45035996273704962,dbadmin,45035996273704974,DATABASE,,VMart,201410-14 11:50:13.230669-04,2014-10-14 11:50:14.06905704,99,5,t,789022736,835955762,10000,174755178
AUDIT_SIZE_BYTES
audit_size_bytes,now,audit
audit_size_bytes,2014-10-14 11:52:14.015231-04,810584417
FLEX_SIZE_BYTES
flex_size_bytes,now,audit_flex
flex_size_bytes,2014-10-14 11:52:15.117036-04,11850
Page 82 of 1217
Administrator's Guide
Configuring the Database
Configuration Parameters
Page 83 of 1217
Administrator's Guide
Configuring the Database
Configuration Procedure
This section describes the tasks required to set up an HP Vertica database. It assumes that you
have obtained a valid license key file, installed the HP Vertica rpm package, and run the installation
script as described in the Installation Guide.
You'll complete the configuration procedure using the:
l
Administration Tools
If you are unfamiliar with Dialog-based user interfaces, read Using the Administration Tools
Interface before you begin. See also the Administration Tools Reference for details.
Note: Users can also perform certain tasks using the Management Console. Those tasks will
point to the appropriate topic.
IMPORTANT NOTES
l
HP strongly recommends that you first use the Tutorial in the Getting Started Guide to
experiment with creating and configuring a database.
Although you may create more than one database (for example, one for production and one for
testing), you may create only one active database for each installation of Vertica Analytic
Database
The generic configuration procedure described here can be used several times during the
development process and modified each time to fit changing goals. You can omit steps such as
preparing actual data files and sample queries, and run the Database Designer without
optimizing for queries. For example, you can create, load, and query a database several times
for development and testing purposes, then one final time to create and load the production
database.
Page 84 of 1217
Administrator's Guide
Configuring the Database
See Also
l
Page 85 of 1217
Administrator's Guide
Configuring the Database
Install HP Vertica
Notes
l
Catalog and data path names must contain only alphanumeric characters and cannot have
leading space characters. Failure to comply with these restrictions will result in database
creation failure.
Page 86 of 1217
Administrator's Guide
Configuring the Database
When you click OK, HP Vertica automatically creates the following subdirectories:
catalog-pathname/database-name/node-name_catalog/data-pathname/database-name/node-name_
data/
For example, if you use the default value (the database administrator's home directory) of
/home/dbadmin for the Stock Exchange example database, the catalog and data directories are
created on each node in the cluster as follows:
/home/dbadmin/Stock_Schema/stock_schema_node1_host01_catalog/home/dbadmin/Stock_
Schema/stock_schema_node1_host01_data
Notes
l
Catalog and data path names must contain only alphanumeric characters and cannot have
leading space characters. Failure to comply with these restrictions will result in database
creation failure.
Page 87 of 1217
Administrator's Guide
Configuring the Database
Isolating execution engine temporary files from data files by creating a separate storage location
for temp space.
Creating labeled storage locations and storage policies, in which selected database objects are
stored on different storage locations based on measured performance statistics or predicted
access patterns.
Each host must have its own catalog and data locations. Hosts cannot share catalog or data
locations.
Configure the storage so that there is enough I/O bandwidth for each node to access the storage
independently.
Page 88 of 1217
Administrator's Guide
Configuring the Database
disk space needed (known as temp space) depends on the nature of the queries, amount of data on
the node and number of concurrent users on the system. By default, any unused disk space on the
data disk can be used as temp space. However, HP recommends provisioning temp space
separate from data disk space. See Configuring Disk Usage to Optimize Performance.
A text editor.
You use data types supported by HP Vertica, as described in the SQL Reference Manual.
Page 89 of 1217
Administrator's Guide
Configuring the Database
Once you have created a database, you can test your schema script by executing it as described in
Create the Logical Schema. If you encounter errors, drop all tables, correct the errors, and run the
script again.
Test data files. Use test files to test the database after the partial data load. If possible, use part
of the actual data files to prepare the test data files.
Actual data files. Once the database has been tested and optimized, use your data files for your
initial Bulk Loading Data.
Large tables
Small tables
HP recommends that you load large tables using multiple files. To test the load process, use files of
10GB to 50GB in size. This size provides several advantages:
l
You can use one of the data files as a sample data file for the Database Designer.
You can load just enough data to Perform a Partial Data Load before you load the remainder.
Page 90 of 1217
Administrator's Guide
Configuring the Database
If a single load fails and rolls back, you do not lose an excessive amount of time.
Once the load process is tested, for multi-terabyte tables, break up the full load in file sizes of
250500GB.
See the Bulk Loading Data and the following additional topics for details:
l
Enforcing Constraints
Tip: You can use the load scripts included in the example databases in the Getting Started
Guide as templates.
Page 91 of 1217
Administrator's Guide
Configuring the Database
Follow with any combination of letters (upper and lowercase), numbers, and/or underscores.
Database names are case sensitive; however, HP strongly recommends that you do not create
databases with the same name that uses different case; for example, do not create a database
called mydatabase and another database called MyDataBase.
Database Passwords
Database passwords may contain letters, digits, and certain special characters; however, no nonASCII Unicode characters may be used. The following table lists special (ASCII) characters that
HP Vertica permits in database passwords. Special characters can appear anywhere within a
password string; for example, mypas$word or $mypassword or mypassword$ are all permitted.
Caution: Using special characters in database passwords that are not listed in the following
table could cause database instability.
Character Description
#
pound sign
exclamation point
plus sign
asterisk
Page 92 of 1217
Administrator's Guide
Configuring the Database
question mark
comma
period
forward slash
equals sign
tilde
minus sign
dollar sign
underscore
colon
space
"
double quote
'
single quote
percent sign
&
ampersand
parenthesis
parenthesis
semicolon
<
>
at sign
back quote
square bracket
square bracket
Page 93 of 1217
Administrator's Guide
Configuring the Database
backslash
caret
vertical bar
curly bracket
curly bracket
See Also
l
Password Guidelines
Page 94 of 1217
Administrator's Guide
Configuring the Database
Database name and password. See Creating a Database Name and Password for rules.
Optionally click Advanced to open the advanced settings and change the port and catalog,
data, and temporary data paths. By default the MC application/web server port is 5450 and
paths are /home/dbadmin, or whatever you defined for the paths when you ran the Cluster
Creation Wizard or the install_vertica script. Do not use the default agent port 5444 as a
new setting for the MC port. See MC Settings > Configuration for port values.
5. Click Continue.
6. Select nodes to include in the database.
The Database Configuration window opens with the options you provided and a graphical
representation of the nodes appears on the page. By default, all nodes are selected to be part of
this database (denoted by a green check mark). You can optionally click each node and clear
Include host in new database to exclude that node from the database. Excluded nodes are
gray. If you change your mind, click the node and select the Include check box.
7. Click Create in the Database Configuration window to create the database on the nodes.
The creation process takes a few moments, after which the database starts and a Success
message appears on the interface.
8. Click OK to close the success message.
MC's Manage page opens and displays the database nodes. Nodes not included in the database
are colored gray, which means they are standby nodes you can include later. To add nodes to or
remove nodes from your HP Vertica cluster, which are not shown in standby mode, you must run
the install_vertica script.
Notes
l
If warnings occur during database creation, nodes will be marked on the UI with an Alert icon
and a message.
Page 95 of 1217
Administrator's Guide
Configuring the Database
Warnings do not prevent the database from being created, but you should address warnings
after the database creation process completes by viewing the database Message Center
from the MC Home page.
Failure messages display on the database Manage page with a link to more detailed
information and a hint with an actionable task that you must complete before you can
continue. Problem nodes are colored red for quick identification.
To view more detailed information about a node in the cluster, double-click the node from the
Manage page, which opens the Node Details page.
To create MC users and grant them access to an MC-managed database, see About MC Users
and Creating an MC User.
See Also
l
Restarting MC
If you are using a remote terminal application, such as PuTTY or a Cygwin bash shell, see
Notes for Remote Terminal Users.
2. Accept the license agreement and specify the location of your license file. For more information
see Managing Licenses for more information.
This step is necessary only if it is the first time you have run the Administration Tools
3. On the Main Menu, click Configuration Menu, and click OK.
4. On the Configuration Menu, click Create Database, and click OK.
5. Enter the name of the database and an optional comment, and click OK.
Page 96 of 1217
Administrator's Guide
Configuring the Database
To provide a password enter the password and click OK. Confirm the password by entering
it again, and then click OK.
If you don't want to provide the password, leave it blank and click OK. If you don't set a
password, HP Vertica prompts you to verify that you truly do not want to establish a
superuser password for this database. Click Yes to create the database without a password
or No to establish the password.
Caution: If you do not enter a password at this point, the superuser password is set to
empty. Unless the database is for evaluation or academic purposes, HP strongly
recommends that you enter a superuser password. See Creating a Database Name and
Password for guidelines.
7. Select the hosts to include in the database from the list of hosts specified when HP Vertica
was installed (install_vertica -s), and click OK.
8. Specify the directories in which to store the data and catalog files, and click OK.
Note: Do not use a shared directory for more than one node. Data and catalog directories
must be distinct for each node. Multiple nodes must not be allowed to write to the same
data or catalog directory.
9. Catalog and data path names must contain only alphanumeric characters and cannot have
leading spaces. Failure to comply with these restrictions results in database creation failure.
For example:
Catalog pathname: /home/dbadmin
Data Pathname: /home/dbadmin
10. Review the Current Database Definition screen to verify that it represents the database you
want to create, and then click Yes to proceed or No to modify the database definition.
11. If you click Yes, HP Vertica creates the database you defined and then displays a message to
indicate that the database was successfully created.
Page 97 of 1217
Administrator's Guide
Configuring the Database
Note: : For databases created with 3 or more nodes, HP Vertica automatically sets Ksafety to 1 to ensure that the database is fault tolerant in case a node fails. For more
information, see Failure Recovery in the Administrator's Guide and MARK_DESIGN_
KSAFE in the SQL Reference Manual.
Page 98 of 1217
Administrator's Guide
Configuring the Database
Page 99 of 1217
Administrator's Guide
Configuring the Database
Note: The database response time depends on factors such as type and size of the application
query, database design, data size and data types stored, available computational power, and
network bandwidth. Adding nodes to a database cluster does not necessarily improve the
system response time for every query, especially if the response time is already short, e.g.,
less then 10 seconds, or the response time is not hardware bound.
Note: HP Vertica requires a dedicated server. If your loader or other processes take up
significant amounts of RAM, it can result in swapping.
Note: Include a sample of queries your users are likely to run against the database. If you
don't have any real queries, just write simple SQL that collects counts on each of your
tables. Alternatively, you can skip this step.
Administrator's Guide
Configuring the Database
Once the database is optimized, it should run queries efficiently. If you discover queries that you
want to optimize, you can modify and update the design. See Incremental Design in the
Administrator's Guide.
See Also
l
COPY
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
Note: If setting the LANG= as shown does not work, the operating system support for
locales may not be installed.
2. For each HP Vertica session (from ODBC/JDBC or vsql) set the language locale.
From vsql:
\locale th_TH
3. From ODBC/JDBC:
"SET LOCALE TO th_TH;"
See Also
l
Supported Locales
About Locales
Administrator's Guide
Configuring the Database
SET LOCALE
For example:
=> ALTERDATABASE mydb SET DefaultSessionLocale = 'en_GB';
<ICU-locale-identifier>;
For example:
=> \locale en_GBINFO:
INFO 2567: Canonical locale: 'en_GB'
Standard collation: 'LEN'
English (United Kingdom)
You can also use the Short Form of a locale in either of these commands:
=>SET LOCALE TO LEN;
Administrator's Guide
Configuring the Database
=>\locale LEN
INFO 2567: Canonical locale: 'en'
Standard collation: 'LEN'
English
You can use these commands to override the locale as many times as needed during a database
session. The session locale setting applies to any subsequent commands issued in the session.
See Also
l
SET LOCALE
Server Locale
The server session locale should be set as described in Specify the Default Locale for the
Database. If you are using different locales in different sessions, at the start of each session from
your client, set the server locale .
vsql Client
l
If thedatabase does not have a default session locale, set the server locale for the session to the
desired locale, as described in Override the Default Locale for a Session.
The locale setting in the terminal emulator where the vsql client runs should be set to be
equivalent to session locale setting on the server side (ICU locale). By doing so, the data is
collated correctly on the server and displayed correctly on the client.
All input data for vsql should be in UTF-8, and all output data is encoded in UTF-8
Administrator's Guide
Configuring the Database
HP Vertica does not support non UTF-8 encodings and associated locale values; .
For instructions on setting locale and encoding, refer to your terminal emulator documentation.
ODBC Clients
l
ODBC applications can be either in ANSI or Unicode mode. If the user application is Unicode,
the encoding used by ODBC is UCS-2. If the user application is ANSI, the data must be in
single-byte ASCII, which is compatible with UTF-8 used on the database server. The ODBC
driver converts UCS-2 to UTF-8 when passing to the HP Vertica server and converts data sent
by the HP Vertica server from UTF-8 to UCS-2.
If the user application is not already in UCS-2, the application must convert the input data to
UCS-2, or unexpected results could occur. For example:
n
For non-UCS-2 data passed to ODBC APIs, when it is interpreted as UCS-2, it could result in
an invalid UCS-2 symbol being passed to the APIs, resulting in errors.
The symbol provided in the alternate encoding could be a valid UCS-2 symbol. If this occurs,
incorrect data is inserted into the database.
If the database does not have a default session locale, ODBC applications should set the
desired server session locale using SQLSetConnectAttr (if different from database wide
setting). By doing so, you get the expected collation and string functions behavior on the server.
JDBC and ADO.NET applications use a UTF-16 character set encoding and are responsible for
converting any non-UTF-16 encoded data to UTF-16. The same cautions apply as for ODBC if
this encoding is violated.
The JDBC and ADO.NET drivers convert UTF-16 data to UTF-8 when passing to the HP
Vertica server and convert data sent by HP Vertica server from UTF-8 to UTF-16.
If there is no default session locale at the database level, JDBC and ADO.NET applications
should set the correct server session locale by executing the SET LOCALE TO command in
order to get the expected collation and string functions behavior on the server. For more
information, see SET LOCALE.
Usage Considerations
Session related:
Administrator's Guide
Configuring the Database
The locale setting is session scoped and applies only to queries (no DML/DDL) executed in that
session. You cannot specify a locale for an individual query.
You can set the default locale for new sessions using the DefaultSessionLocale configuration
parameter
Query related:
The following restrictions apply when queries are run with locale other than the default en_
US@collation=binary:
l
When one or more of the left-side NOT IN columns is CHAR or VARCHAR, multicolumn NOT
IN subqueries are not supported . For example:
=> CREATE TABLE test (x VARCHAR(10), y INT);
=> SELECT ... FROM test WHERE (x,y) NOT IN (SELECT ...);
ERROR: Multi-expression NOT IN subquery is not supported because a left hand
expression could be NULL
Note: Even if columns test.x and test.y have a NOT NULL constraint, an error occurs.
Subqueries that use analytic functions in the HAVING clause are not supported. For example:
=> DROP TABLE test CASCADE;
=> CREATE TABLE test (x VARCHAR(10));
=> SELECT MAX(x)OVER(PARTITION BY 1 ORDER BY 1)
FROM test GROUP BY x HAVING x IN (SELECT MAX(x) FROM test);
ERROR: Analytics query with having clause expression that involves aggregates and
subquery
is not supported
DML/DDL related:
Administrator's Guide
Configuring the Database
SQL identifiers (such as table names and column names) can use UTF-8 Unicode characters.
For example, the following CREATE TABLE statement uses the (German eszett) in the table
name:
=> CREATE TABLE strae(x int, y int);
CREATE TABLE
Projection sort orders are made according to the default en_US@collation=binary collation.
Thus, regardless of the session setting, issuing the following command creates a projection
sorted by col1 according to the binary collation:
=> CREATE PROJECTION p1 AS SELECT * FROM table1 ORDER BY col1;
In such cases, strae and strasse are not stored near each other on disk.
Sorting by binary collation also means that sort optimizations do not work in locales other than
binary. HP Vertica returns the following warning if you create tables or projections in a nonbinary locale:
WARNING: Projections are always created and persisted in the default HP Vertica
locale. The current locale is de_DE
When creating pre-join projections, the projection definition query does not respect the locale or
collation setting. When you insert data into the fact table of a pre-join projection, referential
integrity checks are not locale or collation aware.
For example:
\locale LDE_S1
-- German
=> CREATE TABLE dim (col1 varchar(20) primary key);
=> CREATE TABLE fact (col1 varchar(20) references dim(col1));
=> CREATE PROJECTION pj AS SELECT * FROM fact JOIN dim
ON fact.col1 = dim.col1 UNSEGMENTED ALL NODES;
=> INSERT INTO dim VALUES('');
=> COMMIT;
The following INSERT statement fails with a "nonexistent FK" error even though '' is in the dim
table, and in the German locale 'SS' and '' refer to the same character.
=> INSERT INTO fact VALUES('SS');
ERROR: Nonexistent foreign key value detected in FK-PK join (fact x dim)
Administrator's Guide
Configuring the Database
When the locale is non-binary, HP Vertica uses the COLLATION function to transform the input
to a binary string that sorts in the proper order.
This transformation increases the number of bytes required for the input according to this
formula:
result_column_width = input_octet_width * CollationExpansion + 4
CHAR fields are displayed as fixed length, including any trailing spaces. When CHAR fields are
processed internally, they are first stripped of trailing spaces. For VARCHAR fields, trailing
spaces are usually treated as significant characters; however, trailing spaces are ignored when
sorting or comparing either type of character string field using a non-BINARY locale.
A change to isolation level only applies to future sessions. Existing sessions and their transactions
continue to use the original isolation level.
A transaction retains its isolation level until it completes, even if the session's transaction isolation
Administrator's Guide
Configuring the Database
level changes mid-transaction. HP Vertica internal processes (such as the Tuple Mover and
refresh operations) and DDL operations are always run at SERIALIZABLE isolation level to ensure
consistency.
See Also
l
Transactions
Configuration Parameters
SHOW
Administrator's Guide
Configuring the Database
Configuration Parameters
Configuration parameters are settings that affect database behavior. You can use configuration
parameters to enable, disable, or tune features related to different database aspects like Tuple
Mover, security, Database Designer, or projections. Configuration parameters have default values,
stored in the HP Vertica database. However you can modify certain parameters to configure your
HP Vertica database using one of the following options:
l
From vsql
You can also configure certain parameters at the node level. See Setting and Clearing Configuration
Parameters for more information.
Before you modify a database parameter, review all documentation about the parameter to
determine the context under which you can change it. Some parameter changes do not take effect
until after you restart the database. See the CHANGE_REQUIRES_RESTART column in the
CONFIGURATION_PARAMETERS system table to determine whether a parameter requires a
restart to take effect.
Important: If using 7.1, do not hand edit any vertica.conf files. Additionally, do not use any
workarounds for syncing vertica.conf files.
Administrator's Guide
Configuring the Database
Some settings require that you restart the database, and MC will prompt you to do so. You can
ignore the prompt, but those changes will not take effect until after you restart the database.
If you want to change settings that are specific to Management Console, such as change MC or
agent port assignments, see Managing MC Settings for more information.
See Also
l
Configuration Parameters
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
SHOW CURRENT: Displays active configuration parameter values set at all levels. HP Vertica
checks values set at the session level first. If a value is not set at the session level, HP Vertica
checks whether the value is set for the node where you are logged in. If there is no node-level
setting, HP Vertica checks for a setting at the database level. If no values are set,
SHOWCURRENT shows the default value for the configuration parameter.
SHOW DATABASE: Displays configuration parameter values set for the database.
SHOW SESSION: Displays configuration parameter values set for the current session.
Parameters appear in both system tables if they are configurable at the database, node, and
session levels.
You can set some parameters at the node level. To do so, use the ALTER NODE statement with
the SET parameter:
ALTERNODE node_name SET parameter_name = parameter_value;
Administrator's Guide
Configuring the Database
For example, to prevent clients from connecting to node01, set the MaxClientSessions
configuration parameter to 0:
=> ALTER NODE node01 SET MaxClientSessions = 0;
You can set some parameters at the session level. To do so, use the ALTER SESSION statement
with the SET parameter:
ALTERSESSION SET parameter_name = parameter_value;
For example:
=> ALTERSESSION SET ForceUDxFencedMode = 1;
You can set multiple configuration parameter values at the database, node, and session levels
using comma-separated arguments.
The following example shows how to set multiple parameters at the database level:
ALTER DATABASE mydb SET AnalyzeRowCountInterval = 3600, FailoverToStandbyAfter = '5
minutes';
To clear a node-level parameter value, use the ALTERNODE statement with the CLEAR
parameter. The resulting value reflects the value set at the database level, or the default value if no
database-level value is set.
ALTERNODE node_name CLEAR parameter_name;
In this example, MaxClientSessions will be cleared to the default value (50) on node01:
ALTER NODE node01 CLEAR MaxClientSessions;
To clear a session-level parameter value, use the ALTERSESSION statement with the CLEAR
parameter. If the parameter is set a the node level, the resulting value reflects that setting. If no
Administrator's Guide
Configuring the Database
node-level setting exists, the resulting value reflects the value set at the database level, if
applicable. If the parameter is not set at the node or database level, the resulting value reflects the
default value.
ALTERSESSIONCLEAR parameter_name;
For example:
=> ALTERSESSION CLEAR ForceUDxFencedMode;
You can clear multiple configuration parameter values at the database, node, and session levels
using comma-separated arguments.
The following example shows how to clear multiple parameters at the database level:
ALTER DATABASE mydb CLEAR AnalyzeRowCountInterval, FailoverToStandbyAfter;
General Parameters
You use these general parameters to configure HP Vertica.
Parameters
Description
AnalyzeRowCountInterval
CompressCatalogOnDisk
Administrator's Guide
Configuring the Database
Parameters
Description
CompressNetworkData
EnableCooperativeParse
EnableDataTargetParallelism
EnableResourcePoolCPUAffinity
ExternalTablesExceptionsLimit
FailoverToStandbyAfter
Administrator's Guide
Configuring the Database
Parameters
Description
FencedUDxMemoryLimitMB
FlexTableDataTypeGuessMultiplier
FlexTableRawSize
JavaBinaryForUDx
JavaClassPathForUDx
Administrator's Guide
Configuring the Database
Parameters
Description
MaxAutoSegColumns
MaxClientSessions
PatternMatchAllocator
Administrator's Guide
Configuring the Database
Parameters
Description
PatternMatchStackAllocator
SegmentAutoProjection
TransactionIsolationLevel
TransactionMode
Administrator's Guide
Configuring the Database
Parameters
Description
ActivePartitionCount
MergeOutInterval
MoveOutInterval
MoveOutMaxAgeTime
MoveOutSizePct
The percentage of the WOS that can be filled with data before
the Tuple Mover performs a moveout operation.
Default Value: 0
Example:
ALTERDATABASE mydb SET MoveOutSizePct = 50;
Administrator's Guide
Configuring the Database
Projection Parameters
The following table describes the configuration parameters that help you manage projections.
Parameters
Description
AnalyzeRowCountInterval
EnableGroupByProjections
EnableTopKProjections
EnableExprsInProjections
MaxAutoSegColumns
Administrator's Guide
Configuring the Database
Parameters
Description
SegmentAutoProjection
Description
AdvanceAHMInterval
EpochMapInterval
Administrator's Guide
Configuring the Database
Parameters
Description
HistoryRetentionTime
HistoryRetentionEpochs
Monitoring Parameters
The following table describes the monitoring parameters for configuring HP Vertica.
Administrator's Guide
Configuring the Database
Parameters
Description
SnmpTrapDestinations
List
SnmpTrapsEnabled
SnmpTrapEvents
SyslogEnabled
SyslogEvents
Administrator's Guide
Configuring the Database
Parameters
Description
SyslogFacility
Profiling Parameters
The following table describes the profiling parameters for configuring HP Vertica. See Profiling
Database Performance for more information on profiling queries.
Parameters
Description
GlobalEEProfiling
GlobalQueryProfiling
GlobalSessionProfiling
Security Parameters
The following table describes configuration parameters for configuring client authentication. These
parameters define the hash algorithm, enable SSL, and set the SSL private key, certificate, and
certificate authority for HP Vertica.
Administrator's Guide
Configuring the Database
Parameters
Description
EnableSSL
SSLPrivateKey
Specifies the server's private key. Only the value of this parameter is visible to
dbadmin users.
Default Value: No default value
Example:
ALTERDATABASEmydb SET SSLPrivateKey = '<contents of server.key
file>';
Include the contents of the server.key file, but do not include the file name.
Note: This parameters gets set automatically during upgrade to 7.1 if you
set
EnableSSL=1 prior to the upgrade.
SSLCertificate
Sets the SSL certificate. If your SSL certificate is a certificate chain, cut and
paste only the top-most certificate of the certificate chain to set this value.
Default Value: No default value
Example:
ALTERDATABASEmydb SET SSLCertificate = '<contents of server.crt
file>';
Include the contents of the server.crt file, but do not include the file name.
Note: This parameters gets set automatically during upgrade to 7.1 if you
set
EnableSSL=1 prior to the upgrade.
Administrator's Guide
Configuring the Database
Parameters
Description
SSLCA
Include the contents of the certificate authority root.crt file, but do not
include the file name.
SecurityAlgorit
Sets the algorithm for the function that hash authentication usesMD5 or SHA-
hm
512.
Default Value: 'NONE'
Example:
ALTERDATABASEmydb SET SecurityAlgorithm = 'MD5';
ALTERDATABASEmydb SET SecurityAlgorithm = 'SHA512';
View parameter values with the statement, SHOW DATABASE. You must be a database superuser to
view the value:
SHOW DATABASE mydb SSLCertificate;
See Also
Kerberos Authentication Parameters
Configuring SSL
Administrator's Guide
Configuring the Database
Parameter
Description
DBDCorrelationSampleRowCou
nt
DBDLogInternalDesignProcess
Internationalization Parameters
The following table describes the internationalization parameters for configuring HP Vertica.
Parameters
Description
DefaultIntervalStyle
DefaultSessionLocale
Sets the default session startup locale for the database. This
parameter does not take effect until the database is restarted.
Default Value: en_US@collation=binary
Example:
ALTERDATABASEmydb SET DefaultSessionLocale = 'en_GB';
Administrator's Guide
Configuring the Database
Parameters
Description
EscapeStringWarning
StandardConformingStrings
Description
EnableDataCollector Enables and disables the Data Collector, which is the Workload
Analyzer's internal diagnostics utility. Affects all sessions on all nodes.
Use 0 to turn off data collection.
Default value: 1 (Enabled)
Example:
ALTERDATABASEmydb SET EnableDataCollector = 0;
For more information, see the following topics in the SQL Reference Manual:
Administrator's Guide
Configuring the Database
ANALYZE_WORKLOAD
V_MONITOR.DATA_COLLECTOR
V_MONITOR.TUNING_RECOMMENDATIONS
Analyzing Workloads
Tuning Recommendations
Description
KerberosServiceName Provides the service name portion of the HP Vertica Kerberos principal.
By default, this parameter is 'vertica'. For example:
vertica/[email protected].
KerberosRealm
Administrator's Guide
Configuring the Database
Parameter
Description
KerberosKeytabFile
Provides the location of the keytab file that contains credentials for the
HP Vertica Kerberos principals. By default, this file is located in /etc.
For example: KerberosKeytabFile=/etc/krb5.keytab.
Notes:
l
The keytab file must be readable by the file owner who is running
the process (typically the Linux dbadmin user assigned file
permissions 0600).
KerberosHostname
Description
HCatConnectionTimeout
Administrator's Guide
Configuring the Database
HCatSlowTransferLimit
The lowest transfer speed (in bytes per second) that the HCatalog
Connector allows when retrieving data from the WebHCat server. In
some cases, the data transfer rate from the WebHCat server to HP
Vertica is below this threshold. In such cases, after the number of
seconds specified in the HCatSlowTransferTime parameter pass,
the HCatalog Connector cancels the query and closes the
connection.
Default Value:65536
Requires Restart:No
Example:
ALTERDATABASEmydb SET HCatSlowTransferLimit = 32000;
HCatSlowTransferTime
Note: You can override these configuration parameters when creating an HCatalog schema.
See CREATEHCATALOGSCHEMA in the SQL Reference Manual for an explanation.
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
Many users to access the database without interfering with one another.
Individual schemas can be configured to grant specific users access to the schema and its
tables while restricting others.
Third-party applications to create tables that have the same name in different schemas,
preventing table collisions.
Unlike other RDBMS, a schema in an HP Vertica database is not a collection of objects bound to
one user.
Administrator's Guide
Configuring the Database
through a healthcare management application. HRUser should not be able to access company
healthcare cost information and MedUser should not be able to view personal employee data.
To grant these users access to data they need while restricting them from data they should not
see, two schemas are created with appropriate user access, as follows:
n
Administrator's Guide
Configuring the Database
and the old schema is kept as long as necessary to support the original version of the software.
This is called a rolling application upgrade.
For example, a company might use a HR application to store employee data. The following
schemas could be used for the original and updated versions of the software:
n
HRSchemaA schema owned by HRUser, the schema user for the original HR application.
V2HRSchemaA schema owned by V2HRUser, the schema user for the new version of the
HR application.
IndustrySchemaA schema owned by DBUser (from the retail marketing analytics company)
on which both MartUser and PharmUser have USAGE privileges. It is unlikely that retailers
would be given any privileges beyond USAGE on the schema and SELECT on one or more of its
Administrator's Guide
Configuring the Database
tables.
Creating Schemas
You can create as many schemas as necessary for your database. For example, you could create a
schema for each database user. However, schemas and users are not synonymous as they are in
Oracle.
By default, only a superuser can create a schema or give a user the right to create a schema. (See
GRANT (Database) in the SQL Reference Manual.)
To create a schema use the CREATE SCHEMA statement, as described in the SQL Reference
Manual.
Qualifying the object name by using the schema name and object name separated by a dot. For
example, to specify MyTable, located in Schema1, qualify the name as Schema1.MyTable.
Using a search path that includes the desired schemas when a referenced object is unqualified.
By Setting Search Paths, HP Vertica will automatically search the specified schemas to find the
object.
Administrator's Guide
Configuring the Database
The $user entry in the search path is a placeholder that resolves to the current user name, and
public references the public schema. The v_catalog and v_monitor schemas contain HP
Vertica system tables, and the v_internal schema is for HP Vertica's internal use.
Note: HP Vertica always ensures that the v_catalog, v_monitor, and v_internal schemas are
part of the schema search path.
The default search path has HP Vertica search for unqualified tables first in the users schema,
assuming that a separate schema exists for each user and that the schema uses their user name. If
such a user schema does not exist, or if HP Vertica cannot find the table there, HP Vertica next
search the public schema, and then the v_catalog and v_monitor built-in schemas.
Administrator's Guide
Configuring the Database
A database administrator can set a user's default search schema when creating the user by using
the SEARCH_PATH parameter of the CREATE USER statement. An administrator or the user can
change the user's default search path using the ALTER USER statement's SEARCH_PATH
parameter. Changes made to the default search path using ALTER USER affect new user
sessionsthey do not affect any current sessions.
A user can use the SET SEARCH_PATH statement to override the schema search path for the
current session.
Tip: The SET SEARCH_PATH statement is equivalent in function to the CURRENT_
SCHEMA statement found in some other databases.
To see the current search path, use the SHOWSEARCH_PATH statement. To view the current
schema, use SELECT CURRENT_SCHEMA(). The function SELECT CURRENT_SCHEMA()
also shows the resolved name of $user.
The following example demonstrates displaying and altering the schema search path for the current
user session:
=> SHOW SEARCH_PATH;
name
|
setting
-------------+--------------------------------------------------search_path | "$user", PUBLIC, v_catalog, v_monitor, v_internal
(1 row)
=> SET SEARCH_PATH TO SchemaA, "$user", public;
SET
=> SHOW SEARCH_PATH;
name
|
setting
-------------+-----------------------------------------------------------search_path | SchemaA, "$user", public, v_catalog, v_monitor, v_internal
(1 row)
You can use the DEFAULT keyword to reset the schema search path to the default.
=> SET SEARCH_PATH TO DEFAULT;SET
=> SHOW SEARCH_PATH;
name
|
setting
-------------+--------------------------------------------------search_path | "$user", public, v_catalog, v_monitor, v_internal
(1 row)
To view the default schema search path for a user, query the search_path column of the V_
CATALOG.USERS system table:
=> SELECT search_path from USERS WHERE user_name = 'ExampleUser';
Administrator's Guide
Configuring the Database
search_path
--------------------------------------------------"$user", public, v_catalog, v_monitor, v_internal
(1 row)
=> ALTER USER ExampleUser SEARCH_PATH SchemaA,"$user",public;
ALTER USER
=> SELECT search_path from USERS WHERE user_name = 'ExampleUser';
search_path
-----------------------------------------------------------SchemaA, "$user", public, v_catalog, v_monitor, v_internal
(1 row)
=> SHOW SEARCH_PATH;
name
|
setting
-------------+--------------------------------------------------search_path | "$user", public, v_catalog, v_monitor, v_internal
(1 row)
Note that changing the default search path has no effect ion the user's current session. Even using
the SET SEARCH_PATH DEFAULT statement does not set the search path to the newly-defined
default value. It only has an effect in new sessions.
See Also
l
System Tables
Administrator's Guide
Configuring the Database
Best Practice: When creating objects that span schemas, use qualified table names. This
naming convention avoids confusion if the query path or table structure within the schemas
changes at a later date.
Tables in Schemas
In HP Vertica you can create both base tables and temporary tables, depending on what you are
trying to accomplish. For example, base tables are created in the HP Vertica logical schema while
temporary tables are useful for dividing complex query processing into multiple steps.
For more information, see Creating Tables and Creating Temporary Tables.
If you create a table without providing the projection-related clauses, HP Vertica automatically
creates a superprojection for the table when you use an INSERT INTO or COPY statement to
load data into the table for the first time. The projection is created in the same schema as the
Administrator's Guide
Configuring the Database
table. Once HP Vertica has created the projection, it loads the data.
l
If you use CREATE TABLE AS SELECT to create a table from the results of a query, the table
is created first and a projection is created immediately after, using some of the properties of the
underlying SELECT query.
(Advanced users only) If you use any of the following parameters, the default projection is
created immediately upon table creation using the specified properties:
n
ORDER BY table-column
hash-segmentation-clause
KSAFE
Note: Before you define a superprojection in the above manner, read Creating Custom
Designs in the Administrator's Guide.
See Also
l
Projection Concepts
CREATE TABLE
Administrator's Guide
Configuring the Database
It is a superprojection.
Administrator's Guide
Configuring the Database
Temp tables are not recoverable, so the superprojection is not K-Safe (K-SAFE=0), and you
cannot make it so.
Auto-projections are defined by the table properties and creation methods, as follows:
If table...
Segmentation is...
Same as input
stream (COPY or
stream, if sorted.
INSERT INTO)
the table
Same as input
TABLE AS SELECT
stream, if sorted.
segmented
If not sorted,
sorted using
unsegmented or unknown
query
following rules.
Has FK and PK
FK first, then PK
PK columns
constraints
columns
FK first, then
(no PK)
remaining columns
PK columns
PK columns
On all columns
(no FK)
Has no FK or PK
constraints
Advanced users can modify the default projection created through the CREATE TEMPORARY TABLE
statement by defining one or more of the following parameters:
l
ORDER BY table-column
hash-segmentation-clause
NO PROJECTION
Note: Before you define the superprojection in this manner, read Creating Custom Designs in
Administrator's Guide
Configuring the Database
See Also
l
Projection Concepts
Administrator's Guide
Configuring the Database
Implementing Views
A view is a stored query that dynamically accesses and computes data from the database at
execution time. It differs from a projection in that it is not materialized: it does not store data on
disk. This means that it doesn't need to be refreshed whenever the data in the underlying tables
change, but it does require additional time to access and compute data.
Views are read-only and they support references to tables, temp tables, and other views. They do
not support inserts, deletes, or updates. You can use a view as an abstraction mechanism to:
l
Hide the complexity of SELECT statements from users for support or security purposes. For
example, you could create a view that selects specific columns from specific tables to ensure
that users have easy access to the information they need while restricting them from
confidential information.
Encapsulate the details of the structure of your tables, which could change as your application
evolves, behind a consistent user interface.
Creating Views
A view contains one or more SELECT statements that reference any combination of one or more
tables, temp tables, or views. Additionally, views can specify the column names used to display
results.
The user who creates the view must be a superuser or have the following privileges:
l
SELECT on all the tables and views referenced within the view's defining query.
USAGE on all the schemas that contain the tables and views referenced within the view's
defining query.
To create a view:
1. Use the CREATE VIEW statement to create the view.
2. Use the GRANT (View) statement to grant users the privilege to use the view.
Note: Once created, a view cannot be actively altered. It can only be deleted and recreated.
Administrator's Guide
Configuring the Database
Using Views
Views can be used in the FROM clause of any SQL query or subquery. At execution, HP Vertica
internally substitutes the name of the view used in the query with the actual query used in the view
definition. The following example defines a view (ship) and illustrates how a query that refers to the
view is transformed internally at execution time.
l
New view
=> CREATE VIEW ship AS SELECT * FROM public.shipping_dimension;
Original query
=> SELECT * FROM ship;
Transformed query
=> SELECT * FROM (SELECT * FROM public.shipping_dimension) AS ship;
Tip: To use a view, a user must be granted SELECT permissions on the view. See GRANT
(View).
The following example creates a view named myview that sums all individual incomes of customers
listed in the store.store_sales_fact table by state. The results are grouped in ascending order
by state.
=> CREATE VIEW myview AS
SELECT SUM(annual_income), customer_state
FROM public.customer_dimension
WHERE customer_key IN
(SELECT customer_key
FROM store.store_sales_fact)
GROUP BY customer_state
ORDER BY customer_state ASC;
The following example uses the myview view with a WHERE clause that limits the results to
combined salaries of greater than 2,000,000,000.
=> SELECT * FROM myview where sum > 2000000000;
SUM
| customer_state
-------------+---------------2723441590 | AZ
Administrator's Guide
Configuring the Database
29253817091
4907216137
3769455689
3330524215
4581840709
3310667307
2793284639
5225333668
2128169759
2806150503
2832710696
14215397659
2642551509
(14 rows)
|
|
|
|
|
|
|
|
|
|
|
|
|
CA
CO
CT
FL
IL
IN
MA
MI
NV
PA
TN
TX
UT
The view, however, cannot be used in all queries without generating the same error message. For
example, the following query returns the same error:
=> SELECT * FROM temp;
ERROR: Invalid input for DD: "F"
When you then issue a COUNT command, the returned row count is correct:
=> SELECT COUNT(*) FROM temp;
COUNT
------100
(1 row)
Administrator's Guide
Configuring the Database
This behavior works as intended. You can create views that contain subqueries, where not every
row is intended to pass the predicate.
Administrator's Guide
Configuring the Database
What Is a Design?
A design is a physical storage plan that optimizes query performance. Database Designer uses
sophisticated strategies to create a design that provides excellent performance for ad-hoc queries
and specific queries while using disk space efficiently. Database Designer bases the design on the
following information that you provide:
l
K-safety
Administrator's Guide
Configuring the Database
Design queries: Typical queries that you run during normal database operations. Each query can
be assigned a weight that indicates its relative importance so that Database Designer can
prioritize it when creating the design. Database Designer groups queries that affect the design
that Database Designer creates in the same way and considers one weighted query when
creating a design.
Setting that specifies that Database Designer be guided to create only unsegmented
projections.
Setting that specifies that Database Designer analyze statistics before creating the design.
A design script that creates the projections for the design in a way that meets the optimization
objectives and distributes data uniformly across the cluster.
A deployment script that creates and refreshes the projections for your design. For
comprehensive designs, the deployment script contains commands that remove non-optimized
projections. The deployment script includes the full design script.
A backup script that contains SQLstatements to deploy the design that existed on the system
before deployment. This file is useful in case you need to revert to the pre-deployment design.
While running Database Designer, you can choose to deploy your design automatically after the
deployment script is created, or to deploy it manually, after you have reviewed and tested the
design. HP Vertica recommends that you test the design on a non-production server before
deploying the design to your production server.
Administrator's Guide
Configuring the Database
Recommends buddy projections with the same sort order, which can significantly improve
load, recovery, and site node performance. All buddy projections have the same base name so
that they can be identified as a group.
Note: If you manually create projections, Database Designer recommends a buddy with the
same sort order, if one does not already exist. By default, Database Designer recommends
both super and non-super segmented projections with a buddy of the same sort order and
segmentation.
Accepts a file of sample queries to consider when creating a design. Providing this file is
optional for comprehensive designs.
If you do not provide this file, Database Designer recommends a generic design that does not
consider specific queries. For incremental designs, you must provide sample queries. The query
file can contain up to 100 queries.
Allows you to analyze column correlations. Correlation analysis typically only needs to be
performed once and only if the table has more than DBDCorrelationSampleRowCount (default:
4000) rows.
By default, Database Designer does not analyze column correlations. To set the correlation
analysis mode, use DESIGNER_SET_ANALYZE_CORRELATIONS_MODE
Administrator's Guide
Configuring the Database
Recommends and creates projections in a way that minimizes data skew by distributing data
uniformly across the cluster.
Note: Hewlett-Packard does not recommend live aggregate projections and TopKprojections.
Does not sort, segment, or partition projections on LONG VARBINARY and LONG VARCHAR
columns.
DBADMINuser
Have been assigned the DBDUSERrole and be the owner of database tables for which you are
creating a design
Administrator's Guide
Configuring the Database
3. Grant the user the privilege to create schemas on the database for which they want to create a
design.
=> GRANTCREATE ON DATABASE new_database TO new_user;
5. On all nodes in the cluster, grant the user access to the temporary folder.
=>GRANTALLONLOCATION'/tmp/dbd' TOnew_user;
6. Grant the new user access to the database schema and its tables.
=>GRANTALLONSCHEMAuser_schema TOnew_user;
=> GRANTALLONALLTABLESINSCHEMAuser_schema TOnew_user;
After you have completed this task, you need to do the following to map the MCuser to the new_
user you created in the previous steps:
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
If the user's default role is already DBDUSER, skip this step. Otherwise, The user must
enable the DBDUSER role:
=> SET ROLE DBDUSER;
The DBADMIN must add DBDUSER as the default role for that user:
=> ALTER USER <username> DEFAULT ROLE DBDUSER;
A DBDUSER cannot create a design with a K-safety less than the system K-safety. If the
designs violate the current K-safet by not having enough buddy projections for the tables, the
design does not complete.
A DBDUSER cannot explicitly change the ancient history mark (AHM), even during deployment
of their design.
When you create a design, you automatically have privileges to manipulate the design. Other tasks
may require that the DBDUSER have additional privileges:
To...
Add design tables
Administrator's Guide
Configuring the Database
To...
Add a file of design queries
of a user query
the results of the user query
Create the design and deployment
scripts
The following workflow is common to all these ways to run Database Designer:
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
Projections that Database Designer actually created when the design was deployed
Projections that Database Designer created, but not with the ideal criteria that the Optimizer
identified.
Column optimizations
If you do not deploy the design immediately, review the log to determine if you want to make any
changes. If the design has been deployed, you can still manually create some of the projections
that Database Designer did not create.
To enable the Database Designer logging capability, see Enabling Logging for Database Designer
To view the logged information, see Viewing Database Designer Logs.
Administrator's Guide
Configuring the Database
DC_DESIGN_PROJECTION_CANDIDATES
DC_DESIGN_QUERY_PROJECTION_CANDIDATES
DC_DESIGN_PROJECTION_CANDIDATES
The DC_DESIGN_PROJECTION_CANDIDATES table contains information about all the
projections that the Optimizer proposed. This table also includes the DDL that creates them. The
is_a_winner field indicates if that projection was part of the actual deployed design. To view the
DC_DESIGN_PROJECTION_CANDIDATEStable, enter:
=> SELECT * FROMDC_DESIGN_PROJECTION_CANDIDATES;
DC_DESIGN_QUERY_PROJECTION_CANDIDATES
The DC_DESIGN_QUERY_PROJECTION_CANDIDATEStable lists plan features for all design
queries.
Possible features are:
l
MERGE JOIN
GROUPBY PIPE
RLE PREDICATE
Administrator's Guide
Configuring the Database
LATE MATERIALIZATION
Ideal plan feature and its description, which identifies how the referenced projection should be
optimized.
If the design was deployed, the actual plan feature and its description is included in the table.
This information identifies how the referenced projection was actually optimized.
Because most projections have multiple optimizations, each projection usually has multiple
rows.To view the DC_DESIGN_QUERY_PROJECTION_CANDIDATEStable, enter:
=> SELECT * FROMDC_DESIGN_QUERY_PROJECTION_CANDIDATES;
To see example data from these tables, see Database Designer Logs: Example Data.
Administrator's Guide
Configuring the Database
Administrator's Guide
Configuring the Database
deal_size,
last_deal_update
FROM public.customer_dimension
ORDER BY customer_gender,
annual_income
UNSEGMENTED ALL NODES;
is_a_winner
| t
-[RECORD 2 ]--------+------------------------------------------------------------time
| 2014-04-11 06:30:17.961324-07
node_name
| v_vmart_node0001
session_id
| localhost.localdoma-931:0x1b7
user_id
| 45035996273704962
user_name
| dbadmin
design_id
| 45035996273705182
design_table_id
| 45035996273720624
projection_id
| 45035996273726714
iteration_number
| 1
projection_name
| product_dimension_dbd_2_sort_$product_version$__$product_key$
projection_statement | CREATE PROJECTION v_dbd_sarahtest_sarahtest."product_dimension_
dbd_2_
sort_$product_version$__$product_key$"
(
product_key ENCODING AUTO,
product_version ENCODING RLE,
product_description ENCODING AUTO,
sku_number ENCODING AUTO,
category_description ENCODING AUTO,
department_description ENCODING AUTO,
package_type_description ENCODING AUTO,
package_size ENCODING AUTO,
fat_content ENCODING AUTO,
diet_type ENCODING AUTO,
weight ENCODING AUTO,
weight_units_of_measure ENCODING AUTO,
shelf_width ENCODING AUTO,
shelf_height ENCODING AUTO,
shelf_depth ENCODING AUTO,
product_price ENCODING AUTO,
product_cost ENCODING AUTO,
lowest_competitor_price ENCODING AUTO,
highest_competitor_price ENCODING AUTO,
average_competitor_price ENCODING AUTO,
discontinued_flag ENCODING AUTO
)
AS
SELECT product_key,
product_version,
product_description,
sku_number,
category_description,
department_description,
package_type_description,
package_size,
fat_content,
diet_type,
weight,
weight_units_of_measure,
shelf_width,
Administrator's Guide
Configuring the Database
shelf_height,
shelf_depth,
product_price,
product_cost,
lowest_competitor_price,
highest_competitor_price,
average_competitor_price,
discontinued_flag
FROM public.product_dimension
ORDER BY product_version,
product_key
UNSEGMENTED ALL NODES;
is_a_winner
| f
.
.
.
The next example shows the contents of two records in the DC_DESIGN_QUERY_
PROJECTION_CANDIDATES. Both of these rows apply to projection id 45035996273726626.
In the first record, the Optimizer recommends that Database Designer optimize the customer_
gender column for the GROUPBYPIPEalgorithm.
In the second record, the Optimizer recommends that Database Designer optimize the
public.customer_dimension table for late materialization. Late materialization can improve the
performance of joins that might spill to disk.
=>SELECT * FROMdc_design_query_projection_candidates;
-[RECORD 1 ]-----------------+----------------------------------------------------------time
| 2014-04-11 06:30:17.482377-07
node_name
| v_vmart_node0001
session_id
| localhost.localdoma-931:0x1b7
user_id
| 45035996273704962
user_name
| dbadmin
design_id
| 45035996273705182
design_query_id
| 3
iteration_number
| 1
design_table_id
| 45035996273720620
projection_id
| 45035996273726626
ideal_plan_feature
| GROUP BY PIPE
ideal_plan_feature_description | Group-by pipelined on column(s) customer_gender
dbd_benefits
| 5
opt_path_cost
| 211
-[RECORD 2 ]-----------------+----------------------------------------------------------time
| 2014-04-11 06:30:17.48276-07
node_name
| v_vmart_node0001
session_id
| localhost.localdoma-931:0x1b7
user_id
| 45035996273704962
user_name
| dbadmin
design_id
| 45035996273705182
design_query_id
| 3
Administrator's Guide
Configuring the Database
iteration_number
design_table_id
projection_id
ideal_plan_feature
ideal_plan_feature_description
dbd_benefits
opt_path_cost
.
.
.
|
|
|
|
|
|
|
1
45035996273720620
45035996273726626
LATE MATERIALIZATION
Late materialization on table public.customer_dimension
4
669
You can view the actual plan features that Database Designer implemented for the projections it
created. To do so, query the V_INTERNAL.DC_DESIGN_QUERY_PROJECTIONStable:
=> select * from v_internal.dc_design_query_projections;
-[RECORD 1 ]-------------------+-----------------------------------------------------------time
| 2014-04-11 06:31:41.19199-07
node_name
| v_vmart_node0001
session_id
| localhost.localdoma-931:0x1b7
user_id
| 45035996273704962
user_name
| dbadmin
design_id
| 45035996273705182
design_query_id
| 1
projection_id
| 2
design_table_id
| 45035996273720624
actual_plan_feature
| RLE PREDICATE
actual_plan_feature_description | RLE on predicate column(s) department_description
dbd_benefits
| 2
opt_path_cost
| 141
-[RECORD 2 ]-------------------+-----------------------------------------------------------time
| 2014-04-11 06:31:41.192292-07
node_name
| v_vmart_node0001
session_id
| localhost.localdoma-931:0x1b7
user_id
| 45035996273704962
user_name
| dbadmin
design_id
| 45035996273705182
design_query_id
| 1
projection_id
| 2
design_table_id
| 45035996273720624
actual_plan_feature
| GROUP BY PIPE
actual_plan_feature_description | Group-by pipelined on column(s) fat_content
dbd_benefits
| 5
opt_path_cost
| 155
Administrator's Guide
Configuring the Database
Design Name
Design Types
Optimization Objectives
Design Queries
Statistics Analysis
Design Name
All designs that Database Designer creates must have a name that you specify. The design name
must be alphanumeric or underscore (_)characters, and can be no more than 32 characters long.
(Administrative Tools and Management Console limit the design name to 16 characters.)
The design name becomes part of the files that Database Designer generates, including the
deployment script, allowing the files to be easily associated with a particular Database Designer
run.
Design Types
The Database Designer can create two distinct design types. The design you choose depends on
what you are trying to accomplish:
l
Comprehensive Design
Incremental Design
Comprehensive Design
A comprehensive design creates an initial or replacement design for all the tables in the specified
schemas. Create a comprehensive design when you are creating a new database.
To help Database Designer create an efficient design, load representative data into the tables
before you begin the design process. When you load data into a table, HP Vertica creates an
Administrator's Guide
Configuring the Database
unoptimized superprojection so that Database Designer has projections to optimize. If a table has
no data, Database Designer cannot optimize it.
Optionally, supply Database Designer with representative queries that you plan to use so Database
Designer can optimize the design for them. If you do not supply any queries, Database Designer
creates a generic optimization of the superprojections that minimizes storage, with no queryspecific projections.
During a comprehensive design, Database Designer creates deployment scripts that:
l
Create new projections to optimize query performance, only when they do not already exist.
Create replacement buddy projections when Database Designer changes the encoding of preexisting projections that it has decided to keep.
Incremental Design
An incremental design creates an enhanced design with additional projections, if required, that are
optimized specifically for the queries that you provide. Create an incremental design when you have
one or more queries that you want to optimize.
Optimization Objectives
When creating a design, Database Designer can optimize the design for one of three objectives:
l
Load Database Designer creates a design that is optimized for loads, minimizing database size,
potentially at the expense of query performance.
Performance Database Designer creates a design that is optimized for fast query performance.
Because it recommends a design for optimized query performance, this design might
recommend more than the Load or Balanced objectives, potentially resulting in a larger database
storage size.
Balanced Database Designer creates a design whose objectives are balanced between
database size and query performance.
Administrator's Guide
Configuring the Database
To create the most efficient projections for your database, load a moderate amount of
representative data into tables before running Database Designer. Database Designer considers
the data in this table when creating the design.
If your design tables have a large amount if data, the Database Designer run takes a long time; if
your tables have too little data, the design is not optimized. HP Vertica recommends that 10
GBof sample data is sufficient for creating an optimal design.
If you submit a design table with no data, Database Designer ignores it.
If one of your design tables has been dropped, you will not be able to build or deploy your design.
Design Queries
If you supply representative queries that you run on your database to Database Designer, it
optimizes the performance of those queries.
If you are creating an incremental design, you must supply design queries; if you are creating a
comprehensive design, HP Vertica recommends you supply design queries to create an optimal
design.
Database Designer checks the validity of all queries when you add them to your design and again
when it builds the design. If a query is invalid, Database Designer ignores it.
Query Repository
Using Management Console, you can submit design queries from the QUERY_
REQUESTSsystem table. This is called the query repository.
The QUERY_REQUESTStable contains queries that users have run recently. For a
comprehensive design, you can submit up to 200 queries from the QUERY_REQUESTStable to
Database Designer to be considered when creating the design. For an incremental design, you can
submit up to 100 queries from the QUERY_REQUESTStable.
Administrator's Guide
Configuring the Database
For a comprehensive design, you can make the following changes to the design K-safety before
deploying the design:
l
If your cluster has one or two nodes, you cannot change the K-safety.
If your cluster has three or four nodes, you change the K-safety to 1 or 0.
If you cluster has five or more nodes, you can change the K-safety to 2, 1, or 0.
You cannot change the K-safety value of an incremental design. Incremental designs assume the
K-safety value of your cluster.
K-Safety Requirements
When creating projections with the Database Designer, projection definitions that meet K-Safe
design requirements are recommended and marked with the K-safety level. Note the output from
running the optimized design script generated by the Database Designer in the following example:
=> \i VMart_Schema_design_opt_1.sql
CREATE PROJECTION
CREATE PROJECTION
mark_design_ksafe
---------------------Marked design 1-safe
(1 row)
For more information about K-safety, see K-Safety in the Concepts Guide.
Administrator's Guide
Configuring the Database
Replicated Projections
Unsegmented Projections
Replicated Projections
Replication occurs when HP Vertica stores identical copies of data across all nodes in a cluster.
If you are running on a single-node database, all projections are replicated because segmentation is
not possible in a single-node database.
Assuming that largest-row-count equals the number of rows in the design table with the largest
number of rows, Database Designer recommends that a projection be replicated if any one of the
following is true:
l
Condition 1: largest-row-count < 1,000000 and number of rows in the table <= 10% of largestrow-count.
Condition 2: largest-row-count >= 10,000,000 and number of rows in the table <= 1% of largestrow-count.
For more information about replication, see High Availability Through Projections in the Concepts
Guide.
Unsegmented Projections
Segmentation occurs when HP Vertica distributes data evenly across multiple database nodes so
that all nodes participate in query execution. Projection segmentation provides high availability and
recovery, and optimizes query execution.
When running Database Designer programmatically or using Management Console, you can
specify to allow Database Designer to recommend unsegmented projections in the design. If you
do not specify this, Database Designer recommends only segmented projections.
Database Designer recommends segmented superprojections for large tables when deploying to
multiple node clusters, and recommends replicated superprojections for smaller tables.
Database Designer does not segment projections on:
l
Single-node clusters
Administrator's Guide
Configuring the Database
For more information about segmentation, see High Availability Through Projections in the
Concepts Guide.
Statistics Analysis
By default, Database Designer analyzes statistics for the design tables when adding them to the
design. This option is optional, but HPHP Vertica recommends that you analyze statistics because
accurate statistics help Database Designer optimize compression and query performance.
Analyzing statistics takes time and resources. If the current statistics for the design tables are up to
date, do not bother analyzing the statistics. When in doubt, analyze the statistics to make sure they
are current.
For more information, see Collecting Statistics.
Building a Design
After you have created design tables and loaded data into them, and then specified the parameters
you want Database Designer to use when creating the physical schema, direct Database Designer
to create the scripts necessary to build the design.
Note: You cannot stop a running database if Database Designer is building a database design.
When you build a database design, HP Vertica generates two scripts:
l
If you run Database Designer from Administrative Tools,.HP HP Vertica also creates a backup
script named <design_name>_projection_backup_<unique id #>.sql. This script contains
SQLstatements to deploy the design that existed on the system before deployment. This file is
useful in case you need to revert to the pre-deployment design.
When you create a design using Management Console:
Administrator's Guide
Configuring the Database
If you submit a large number of queries to your design and build it right immediately, a timing
issue could cause the queries not to load before deployment starts. If this occurs, you may see
one of the following errors:
n
To accommodate this timing issue, you may need to reset the design, check the Queries tab to
make sure the queries have been loaded, and then rebuild the design. Detailed instructions are
in:
The scripts are deleted when deployment completes. To save a copy of the deployment script
after the design is built but before the deployment completes, go to the Output window and copy
and paste the SQLstatements to a file.
Resetting a Design
You must reset a design when:
l
You build a design and the output scripts described in Building a Design are not created.
You build a design but Database Designer cannot complete the design because the queries it
expects are not loaded.
Resetting a design discards all the run-specific information of the previous Database Designer
build, but retains its configuration (design type, optimization objectives, K-safety, etc.) and tables
and queries.
After you reset a design, review the design to see what changes you need to make. For example,
you can fix errors, change parameters, or check for and add additional tables or queries. Then you
can rebuild the design.
You can only reset a design in Management Console or by using the DESIGNER_RESET_
DESIGNfunction.
Administrator's Guide
Configuring the Database
Deploying a Design
After running Database Designer to generate a deployment script, HP Vertica recommends that
you test your design on a non-production server before you deploy it to your production server.
Both the design and deployment processes run in the background. This is useful if you have a large
design that you want to run overnight. Because an active SSH session is not required, the
design/deploy operations continue to run uninterrupted, even if the session is terminated.
Note: You cannot stop a running database if Database Designer is building or deploying a
database design.
Database Designer runs as a background process. Multiple users can run Database Designer
concurrently without interfering with each other or using up all the cluster resources. However, if
multiple users are deploying a design on the same tables at the same time, Database Designer may
not be able to complete the deployment. To avoid problems, consider the following:
l
Avoid scheduling Database Designer runs on the same set of tables at the same time.
Administrator's Guide
Configuring the Database
If you choose not to have Database Designer automatically run the deployment script (for example,
if you want to maintain projections from a pre-existing deployment), you can manually run the
deployment script later. See Deploying Designs Manually.
To deploy a design while running Database Designer, do one of the following:
l
In the Administration Tools, select Deploy design in the Design Options window.
From Management Console, open a database and select the Design page at the bottom of the
window.
For details about using Management Console to create a design, see Using Management
Console to Create a Design
Administrator's Guide
Configuring the Database
From the Administration Tools menu, by selecting Configuration Menu > Run Database
Designer. You must be a DBADMIN user to run Database Designer from the Administration
Tools.
For details about using Administration Tools to create a design, see Using Administration Tools
to Create a Design.
The following table shows what Database Designer capabilities are available in each tool:
Database Designer
Capability
Management
Console
Create design
Design name length (# of
Running Database
Designer
Programmatically
Administrative
Tools
Yes
Yes
Yes
16
32
16
Yes
Yes
Yes
characters)
Build design (create design
and deployment scripts)
Create backup script
Set design type
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
(comprehensive or
incremental)
Yes
Yes
Yes
Set K-safety
Yes
Yes
Yes
Analyze statistics
Yes
Yes
Yes
Administrator's Guide
Configuring the Database
Database Designer
Capability
Management
Console
Running Database
Designer
Programmatically
Yes
Yes
Yes
Yes
Administrative
Tools
projections
View event history
Set correlation analysis
Yes
mode (Default = 0)
and Next to navigate through the Wizard steps, or Cancel to cancel creating a new design.
To learn how to use the Wizard to create a design, see Using the Wizard to Create a Design.
l
ManualThis option creates and saves a design with the default parameters.
To learn how to create a design manually, see Creating a Design Manually
Tip: If you have many design tables that you want Database Designer to consider, it might be
easier to use the Wizard to create your design. In the Wizard, you can submit all the tables in a
schema at once; creating a design manually requires that you submit the design tables one at a
time.
Administrator's Guide
Configuring the Database
The left side of the Database Designer window lists the database designs you own, with the
most recent design you worked on selected. That pane also lists the current status of the
design.
The main pane contains details about the selected design.
Administrator's Guide
Configuring the Database
5. To build the design immediately after exiting the Wizard, on the Execution Options window,
select Auto-build.
Important: Hewlett-Packard does not recommend that you auto-deploy the design from
the Wizard. There may be a delay in adding the queries to the design, so if the design is
deployed but the queries have not yet loaded, deployment may fail. If this happens, reset
the design, check the Queries tab to make sure the queries have been loaded, and deploy
the design.
6. When you have entered all the information, the Wizard displays a summary of your choices.
Click Submit Design to build your design.
Administrator's Guide
Configuring the Database
1. Log in to Management Console, select and start your database, and click Design at the bottom
of the window. The Database Designer window appears.
The left side of the Database Designer window lists the database designs you own, with the
most recent design you worked on highlighted. That pane also lists the current status of the
design. Details about the most recent design appear in the main pane.
The main pane contains details about the selected design.
Administrator's Guide
Configuring the Database
After a few seconds, the main Database Design window opens, displaying the default design
parameters. HP Vertica has created and saved a design with the name you specified, and
assigned it the default parameters.
For more information, see Design Name.
4. On the General window, modify the design type, optimization objectives, K-safety, Analyze
Correlations Mode, and the setting that allows Database Designer to create unsegmented
projections.
If you choose Incremental, the design automatically optimizes for the desired queries, and the
K-safety defaults to the value of the cluster K-safety; you cannot change these values for an
incremental design.
Analyze Correlations Mode determines if Database Designer analyzes and considers column
correlations when creating the design. For more information, see DESIGNER_SET_
ANALYZE_CORRELATIONS_MODE.
5. Click the Tables tab. You must submit tables to your design.
6. To add tables of sample data to your design, click AddTables. A list of available tables
appears; select the tables you want and click Save. If you want to remove tables from your
design, click the tables you want to remove, and click Remove Selected.
If a design table has been dropped from the database, a red circle with a white exclamation
point appears next to the table name. Before you can build or deploy the design, you must
remove any dropped tables from the design. To do this, select the dropped tables and and click
Remove Selected. You cannot build or deploy a design if any of the design tables have been
dropped.
7. Click the Queries tab. To add queries to your design, do one of the following:
n
To add queries from a file, select Choose File. All valid queries in the file that you select are
added to the design and appear in the Queries window.
Database Designer checks the validity of the queries when you add the queries to the design
and again when you build the design. If it finds invalid queries, it ignores them.
Administrator's Guide
Configuring the Database
If you have a large number of queries, it may take time to load them. Make sure that all the
queries you want Database Designer to consider when creating the design are listed in the
Queries window.
8. Once you have specified all the parameters for your design, you should build the design. To do
this, select your design and click Build Design.
9. Select Analyze Statistics if you want Database Designer to analyze the statistics before
building the design.
For more information see Statistics Analysis.
10. If you do not need to review the design before deploying it, select Deploy Immediately.
Otherwise, leave that option unselected.
11. Click Start. On the left-hand pane, the status of your design displays as Building until it is
complete.
12. To follow the progress of a build, click Event History. Status messages appear in this window
and you can see the current phase of the build operation. The information in the Event History
tab contains data from the OUTPUT_EVENT_HISTORY system table.
13. When the build completes, the left-hand pane displays Built. To view the deployment script,
select your design and click Output.
14. After you deploy the design using Management Console, the deployment script is deleted. To
keep a permanent copy of the deployment script, copy and paste the SQLcommands from the
Output window to a file.
15. Once you have reviewed your design and are ready to deploy it, select the design and click
Deploy Design.
16. To follow the progress of the deployment, click Event History. Status messages appear in this
window and you can see the current phase of the deployment operation.
In the Event History window, while the design is running, you can do one of the following:
n
Click the blue button next to the design name to refresh the event history listing.
Click Force Delete Design to cancel and delete the design in progress.
Administrator's Guide
Configuring the Database
17. When the deployment completes, the left-hand pane displays Deployment Completed. To
view the deployment script, select your design and click Output.
Your database is now optimized according to the parameters you set.
Administrator's Guide
Configuring the Database
10. On the Optimization Objectives window, select the objective you want for the database
optimization:
Optimize with Queries
Update statistics
For more information see Statistics Analysis.
Deploy design
For more information, see Deploying a Design.
Proceed with building the design, and deploying it if you specified to deploy it immediately.
If you did not specify to deploy, you can review the design and deployment scripts and
deploy them manually, as described in Deploying Designs Manually.
Cancel the design and go back to change some of the parameters as needed.
12. Creating a design can take a long time.To cancel a running design from the Administration
Tools window, enter Ctrl+C.
To create a design for the VMart example database, see Using Database Designer to Create a
Comprehensive Design in the Getting Started Guide.
Set the optimization objective to prioritize for query performance or storage footprint.
Create the script that contains the DDL statements that create the design projections.
Important: When you grant the DBDUSER role, you must associate a resource pool with that
user to manage resources during Database Designer runs. Multiple users can run Database
Designer concurrently without interfering with each other or using up all the cluster resources.
When a user runs Database Designer, either using the Administration Tools or
programmatically, its execution is mostly contained by the user's resource pool, but may spill
over into some system resource pools for less-intensive tasks.
Administrator's Guide
For detailed information about each function, see Database Designer Functions in the SQL
Reference Manual.
Add individual queries and tables, or add data to your design and then rerun Database Designer
to update the design based on this new information.
Use recently executed queries to set up your database to run Database Designer automatically
on a regular basis.
Assign each design query a query weight that indicates the importance of that query in creating
the design. Assign a higher weight to queries that you run frequently so that Database Designer
prioritizes those queries in creating the design.
Setup Functions
This function directs Database Designer to create a new design:
l
DESIGNER_CREATE_DESIGN
Configuration Functions
The following functions allow you to specify properties of a particular design:
l
DESIGNER_DESIGN_PROJECTION_ENCODINGS
DESIGNER_SET_DESIGN_KSAFETY
DESIGNER_SET_OPTIMIZATION_OBJECTIVE
Administrator's Guide
DESIGNER_SET_DESIGN_TYPE
DESIGNER_SET_PROPOSED_UNSEGMENTED_PROJECTIONS
DESIGNER_SET_ANALYZE_CORRELATIONS_MODE
Input Functions
The following functions allow you to add tables and queries to your Database Designer design:
l
DESIGNER_ADD_DESIGN_QUERIES
DESIGNER_ADD_DESIGN_QUERIES_FROM RESULTS
DESIGNER_ADD_DESIGN_QUERY
DESIGNER_ADD_DESIGN_TABLES
Invocation Functions
These functions populate the Database Designer workspace and create design and deployment
scripts. You can also analyze statistics, deploy the design automatically, and drop the workspace
after the deployment:
l
DESIGNER_RUN_POPULATE_DESIGN_AND_DEPLOY
DESIGNER_WAIT_FOR_DESIGN
Output Functions
The following functions display information about projections and scripts that the Database
Designer created:
l
DESIGNER_OUTPUT_ALL_DESIGN_PROJECTIONS
DESIGNER_OUTPUT_DEPLOYMENT_SCRIPT
Cleanup Functions
The following functions cancel any running Database Designer operation or drop a Database
Designer design and all its contents:
Administrator's Guide
DESIGNER_CANCEL_POPULATE_DESIGN
DESIGNER_DROP_DESIGN
DESIGNER_DROP_ALL_DESIGNS
Administrator's Guide
A DBDUSER can change K-safety for their own designs, but they cannot change the system Ksafety value. The DBDUSER can set the K-safety to a value less than or equal to the system Ksafety value, but is limited to a value of 0, 1, or 2.
A DBDUSER cannot explicitly change the ancient history mark (AHM), even during deployment
of their design.
DBDUSER Privileges
When you create a design, you automatically have privileges to manipulate the design. Other tasks
may require that the DBDUSER have additional privileges:
To...
Add tables to a design
design
Add a query file to the design
scripts
Administrator's Guide
Note: Be sure to back up the existing design using the EXPORT_CATALOG (on page 1)
function before running the Database Designer functions on an existing schema. You must
explicitly back up the current design when using Database Designer to create a new
comprehensive design.
Before you run this example, you should have the DBDUSER role, and you should have enabled
that role using the SET ROLE DBDUSER command:
1. Create a table in the public schema:
=> CREATE TABLE T(
x INT,
y INT,
z INT,
u INT,
v INT,
w INT PRIMARY KEY
);
4. Copy the data from table T1 to table T2 and commit the changes:
=> INSERT /*+DIRECT*/ INTO T2 SELECT * FROM T;
=> COMMIT;
Administrator's Guide
This command adds information to the DESIGNS system table in the V_MONITOR schema.
6. Add tables from the public schema to the design :
=> SELECT DESIGNER_ADD_DESIGN_TABLES('my_design', 'public.t');
=> SELECT DESIGNER_ADD_DESIGN_TABLES('my_design', 'public.t2');
8. Add the queries file to the design and display the resultsthe numbers of accepted queries,
non-design queries, and unoptimizable queries:
=> SELECT DESIGNER_ADD_DESIGN_QUERIES
('my_design',
'/tmp/examples/queries.txt',
'true'
);
of
of
of
of
accepted queries
=2
queries referencing non-design tables =0
unsupported queries
=0
illegal queries
=0
Administrator's Guide
10. Set the optimization objective to query. This setting creates a design that focuses on faster
query performance, which might recommend additional projections. These projections could
result in a larger database storage footprint:
=> SELECT DESIGNER_SET_OPTIMIZATION_OBJECTIVE('my_design', 'query');
11. Create the design and save the design and deployment scripts in /tmp/examples, or another
directory where you have READ and WRITE privileges. The following command:
n
Analyzes statistics
DEPLOYMENT_PROJECTION_STATEMENTS
DEPLOYMENT_PROJECTIONS
OUTPUT_DEPLOYMENT_STATUS
12. Examine the status of the Database Designer run to see what projections Database Designer
recommends. In the deployment_projection_name column:
n
Administrator's Guide
13. View the script /tmp/examples/my_design_deploy.sql to see how these projections are
created when you run the deployment script. In this example, the script also assigns the
encoding schemes RLE and COMMONDELTA_COMP to columns where appropriate.
14. Deploy the design from the directory where you saved it:
=> \i /tmp/examples/my_design_deploy.sql
Administrator's Guide
Administrator's Guide
Become familiar with standard design requirements and plan your design to include them. See
Design Requirements.
Determine how many projections you need to include in the design. See Determining the
Number of Projections to Use.
Determine the type of compression and encoding to use for columns. See Data Encoding and
Compression.
Determine whether or not you want the database to be K-safe. HP Vertica recommends that all
production databases have a minimum K-safety of one (K=1). Valid K-safety values are 0, 1, and
2. See Designing for K-Safety.
Design Requirements
A physical schema design is a script that contains CREATE PROJECTION statements. These
statements determine which columns are included in projections and how they are optimized.
If you use Database Designer as a starting point, it automatically creates designs that meet all
fundamental design requirements. If you intend to create or modify designs manually, be aware that
all designs must meet the following requirements:
l
Every design must create at least one superprojection for every table in the database that is
used by the client application. These projections provide complete coverage that enables users
to perform ad-hoc queries as needed. They can contain joins and they are usually configured to
maximize performance through sort order, compression, and encoding.
Query-specific projections are optional. If you are satisfied with the performance provided
through superprojections, you do not need to create additional projections. However, you can
maximize performance by tuning for specific query work loads.
HP recommends that all production databases have a minimum K-safety of one (K=1) to support
high availability and recovery. (K-safety can be set to 0, 1, or 2.) See High Availability Through
Projections in the Concepts Guide and Designing for K-Safety.
Administrator's Guide
HP Vertica recommends that if you have more than 20 nodes, but small tables, do not create
replicated projections. If you create replicated projections, the catalog becomes very large and
performance may degrade. Instead, consider segmenting those projections.
The amount of disk space you have available on each node in the cluster
The amount of time available for loading data into the database
As the number of projections that are tuned for specific queries increases, the performance of these
queries improves. However, the amount of disk space used and the amount of time required to load
data increases as well. Therefore, you should create and test designs to determine the optimum
number of projections for your database configuration. On average, organizations that choose to
implement query-specific projections achieve optimal performance through the addition of a few
query-specific projections.
Administrator's Guide
1+
3+
5+
2K+1
Monitoring K-Safety
Monitoring tables can be accessed programmatically to enable external actions, such as alerts.
You monitor the K-safety level by polling the SYSTEM table column and checking the value. See
SYSTEM in the SQL Reference Manual.
Loss of K-Safety
When K nodes in your cluster fail, your database continues to run, although performance is
affected. Further node failures could potentially cause the database to shut down if the failed node's
data is not available from another functioning node in the cluster.
Administrator's Guide
You can modify a design created for a three-node (or greater) cluster, and the K-safe requirements
are already set.
If you create custom projections, your physical schema design must meet the following
requirements to be able to successfully recover the database in the event of a failure:
l
Segmented projections must be segmented across all nodes. Refer to Designing for
Segmentation and Designing Segmented Projections for K-Safety.
Replicated projections must be replicated on all nodes. See Designing Replicated Projections
for K-Safety.
Segmented projections must have K buddy projections (projections that have identical columns
and segmentation criteria, except that corresponding segments are placed on different nodes).
You can use the MARK_DESIGN_KSAFE function to find out whether your schema design meets
requirements for K-safety.
Create a segmented projection for each fact and large dimension table.
Create segmented buddy projections for each of these projections. The total number of
projections in a buddy set must be two for a K=1 database or three for a K=2 database.
For an overview of segmented projections and their buddies, see Projection Segmentation in the
Concepts Guide. For information about designing for K-safety, see Designing for K-Safety and
Designing for Segmentation.
Administrator's Guide
Segmenting Projections
To segment a projection, use the segmentation clause to specify the:
l
Nodes on which to segment the projection. You can segment projections across all the nodes, or
just the number of nodes necessary to maintain K-safety, either three for a K=1 database or five
for a K=2 database.
Rename it to something similar to the name of the original projection. For example, a projection
named retail_sales_fact_P1 could have buddies named retail_sales_fact_P1_B1 and
retail_sales_fact_P1_B2.
Create an offset to store the segments for the buddy on different nodes. For example, the first
buddy in a projection set would have an offset of one (OFFSET1;) the second buddy in a
projection set would have an offset of two (OFFSET2;), and so on.
Administrator's Guide
If, for example, the nodes are named NODE01, NODE02, and NODE03, the projections are named
ABC_NODE01, ABC_NODE02, and ABC_NODE03.
Note: This naming convention can affect functions that provide information about projections,
for example, GET_PROJECTIONS or GET_PROJECTION_STATUS, where you must
provide the name ABC_NODE01 instead of just ABC. To view a list of the nodes in a database,
use the View Database command in the Administration Tools.
The following script uses the UNSEGMENTED ALL NODES syntax to create one unsegmented
superprojection for the store_dimension table on each node.
CREATE PROJECTION store_dimension( C0_store_dimension_floor_plan_type ENCODING RLE ,
C1_store_dimension_photo_processing_type ENCODING RLE ,
C2_store_dimension_store_key ,
C3_store_dimension_store_name ,
C4_store_dimension_store_number ,
C5_store_dimension_store_street_address ,
C6_store_dimension_store_city ,
C7_store_dimension_store_state ,
C8_store_dimension_store_region ,
C9_store_dimension_financial_service_type ,
C10_store_dimension_selling_square_footage ,
C11_store_dimension_total_square_footage ,
C12_store_dimension_first_open_date ,
C13_store_dimension_last_remodel_date )
AS SELECT T_store_dimension.floor_plan_type,
T_store_dimension.photo_processing_type,
T_store_dimension.store_key,
T_store_dimension.store_name,
T_store_dimension.store_number,
T_store_dimension.store_street_address,
T_store_dimension.store_city,
T_store_dimension.store_state,
Administrator's Guide
T_store_dimension.store_region,
T_store_dimension.financial_service_type,
T_store_dimension.selling_square_footage,
T_store_dimension.total_square_footage,
T_store_dimension.first_open_date,
T_store_dimension.last_remodel_date
FROM store_dimension T_store_dimension
ORDER BY T_store_dimension.floor_plan_type, T_store_dimension.photo_processing_type
UNSEGMENTED ALL NODES;
Note: Large dimension tables can be segmented. A dimension table is considered to be large
when it is approximately the same size as a fact table.
Administrator's Guide
Design Fundamentals
Although you can write custom projections from scratch, HP Vertica recommends that you use
Database Designer to create a design to use as a starting point. This ensures that you have
projections that meet basic requirements.
Administrator's Guide
Designing Superprojections
Superprojections have the following requirements:
l
For a K-safe design, superprojections must either be replicated on all nodes within the database
cluster (for dimension tables) or paired with buddies and segmented across all nodes (for very
large tables and medium large tables). See Physical Schema and High Availability Through
Projections in the Concepts Guide for an overview of projections and how they are stored. See
Designing for K-Safety for design specifics.
Minimizing StorageRequirements
Minimizing storage not only saves on physical resources, it increases performance by requiring the
database to perform less disk I/O. To minimize storage space for a projection:
l
Analyze the type of data stored in each projection column and choose the most effective
encoding method. See the CREATE PROJECTION statement and encoding-type in the SQL
Reference Manual.
The HP Vertica optimizer gives Run-Length Encoding (RLE) preference, so be sure to use it
whenever appropriate. Run Length Encoding (RLE) replaces sequences (runs) of identical
values with a single pair that contains the value and number of occurrences. Therefore, use it
only when the run length is large, such as when sorting low-cardinality columns.
Prioritize low-cardinality columns in the column sort order. This minimizes the number of rows
that HP Vertica stores and accesses to retrieve query results.
For more information about minimizing storage requirements, see Choosing Sort Order: Best
Practices.
Administrator's Guide
columns that are actually used in queries. For examples that take into account both storage and
query requirements, see Choosing Sort Order: Best Practices.
Note: For maximum performance, do not sort projections on LONG VARBINARY and LONG
VARCHAR columns.
Projections within a buddy set can all have different sort orders. This enables you to maximize
query performance for groups of queries with common WHERE clauses, but different sort orders.
If, for example, you have a three-node cluster, your buddy set contains three interrelated
projections, each having its own sort order.
In a database with a K-safety of 1 or 2, buddy projections are used for data recovery. If a node fails,
it queries the other nodes to recover data through buddy projections. If a projection's buddies use
different sort orders, it takes longer to recover the projection because the data has to be resorted
during recovery to match the sort order of the projection.
To address this, consider using identical sort orders for tables that are rarely queried or that are
repeatedly accessed by the same query, and use multiple sort orders for tables that are accessed
by queries with common WHERE clauses, but different sort orders.
If you have queries that access multiple tables or you want to maintain the same sort order for
projections within buddy sets, create query-specific projections. Designs that contain projections
for specific queries are called optimized designs.
Administrator's Guide
HP Vertica can use a local merge join if tables target and source use one of the following
projection designs, where their inputs are pre-sorted through the CREATE PROJECTION ORDER BY
clause:
l
Tip: For best merge performance, the source table should be smaller than the target table.
See Also
l
Administrator's Guide
# of Distinct Values
Encoded With
gender
2 (M or F)
RLE
pass_fail
2 (P or F)
RLE
class
name
Auto
The fastest way to access the data is to work through the low-cardinality columns with the smallest
number of distinct values before the high-cardinality columns. The following sort order minimizes
storage and maximizes query performance for queries that have equality restrictions on gender,
class, pass_fail, and name. Specify the ORDER BY clause of the projection as follows:
ORDER BY students.gender, students.pass_fail, students.class, students.name
In this example, the gender column is represented by two RLE entries, the pass_fail column is
represented by four entries, and the class column is represented by 16 entries, regardless of the
cardinality of the students table. HP Vertica efficiently finds the set of rows that satisfy all the
predicates, resulting in a huge reduction of search effort for RLE encoded columns that occur early
in the sort order. Consequently, if you use low-cardinality columns in local predicates, as in the
Administrator's Guide
previous example, put those columns early in the projection sort order, in increasing order of distinct
cardinality (that is, in increasing order of the number of distinct values in each column).
If you sort this table with student.class first, you improve the performance of queries that restrict
only on the student.class column, and you improve the compression of the student.class
column (which contains the largest number of distinct values), but the other columns do not
compress as well.Determining which projection is better depends on the specific queries in your
workload, and their relative importance.
Storage savings with compression decrease as the cardinality of the column increases; however,
storage savings with compression increase as the number of bytes required to store values in that
column increases.
The zipcode column might not have 10 sorted entries in a row with the same zip code, so there is
probably no advantage to run-length encoding that column, and it could make compression worse.
But there are likely to be more than 10 countries in a sorted run length, so applying RLE to the
country column can improve performance.
Administrator's Guide
In the query, put the Area_Code column first, so that only the values in the Number column that start
with 978 are scanned.
=>SELECT AddressFROM customer_info WHERE Area_Code='978' AND Number='9780123457';
Merge joinIf both inputs are pre-sorted on the join column, the optimizer chooses a merge
join, which is faster and uses less memory.
Administrator's Guide
Hash joinUsing the hash join algorithm, HP Vertica uses the smaller (inner) joined table to
build an in-memory hash table on the join column. A hash join has no sort requirement, but it
consumes more memory because Vertica builds a hash table with the values in the inner table.
The optimizer chooses a hash join when projections are not sorted on the join columns.
If both inputs are pre-sorted,merge joins do not have to do any pre-processing, making the join
perform faster. HP Vertica uses the term sort-merge join to refer to the case when at least one of
the inputs must be sorted prior to the merge join. HP Vertica sorts the inner input side but only if the
outer input side is already sorted on the join columns.
To give the Vertica query optimizer the option to use an efficient merge join for a particular join,
create projections on both sides of the join that put the join column first in their respective
projections. This is primarily important to do if both tables are so large that neither table fits into
memory. If all tables that a table will be joined to can be expected to fit into memory simultaneously,
the benefits of merge join over hash join are sufficiently small that it probably isn't worth creating a
projection for any one join column.
Administrator's Guide
In the following example, the table inventory stores inventory data, and product_key and
warehouse_key are foreign keys to the product_dimension and warehouse_dimension tables:
=> CREATE TABLE inventory (
date_key INTEGER NOT NULL,
product_key INTEGER NOT NULL,
warehouse_key INTEGER NOT NULL,
...
);
=> ALTER TABLE inventory
ADD CONSTRAINT fk_inventory_warehouse FOREIGN KEY(warehouse_key)
REFERENCES warehouse_dimension(warehouse_key);
ALTER TABLE inventory
ADD CONSTRAINT fk_inventory_product FOREIGN KEY(product_key)
REFERENCES product_dimension(product_key);
The inventory table should be sorted by warehouse_key and then product, since the cardinality of
the warehouse_key column is probably lower that the cardinality of the product_key.
Columns in the sort order are given the highest priority (numbers > 1000).
The last column in the sort order is given the rank number 1001.
The next-to-last column in the sort order is given the rank number 1002, and so on until the first
column in the sort order is given 1000 + # of sort columns.
The remaining columns are given numbers from 10001, starting with 1000 and decrementing by
one per column.
HP Vertica then stores columns on disk from the highest ranking to the lowest ranking. It places
highest-ranking columns on the fastest disks and the lowest-ranking columns on the slowest disks.
Overriding Default Column Ranking
You can modify which columns are stored on fast disks by manually overriding the default ranks for
these columns. To accomplish this, set the ACCESSRANK keyword in the column list. Make sure to
Administrator's Guide
use an integer that is not already being used for another column. For example, if you want to give a
column the fastest access rank, use a number that is significantly higher than 1000 + the number of
sort columns. This allows you to enter more columns over time without bumping into the access
rank you set.
The following example sets the access rank for the C1_retail_sales_fact_store_key column to
1500.
CREATE PROJECTION retail_sales_fact_P1 ( C1_retail_sales_fact_store_key ENCODING RLE
ACCESSRANK 1500,
C2_retail_sales_fact_pos_transaction_number ,
C3_retail_sales_fact_sales_dollar_amount ,
C4_retail_sales_fact_cost_dollar_amount )
Administrator's Guide
Projection Examples
This section provides examples that show you how to create projections.
Segmented projections
To support K-safety=2, the database requires three segmented projections (one projection and
two buddy projections) for each fact table. In this case, it requires three segmented projections
for the retail_sale_fact table:
Projection Description
P1
P1_B1
The first buddy projection for P1. This buddy is required to provide K-safety=1.
P1_B2
The second buddy projection for P1. This buddy is required to provide Ksafety=2.
Unsegmented Projections
To support the database, one unsegmented superprojection must be created for each dimension
table on each node. In this case, one unsegmented superprojection must be created on each
node for the store_dimension table:
Node
Unsegmented Projection
Node01 store_dimension_Node01
Node02 store_dimension_Node02
Node03 store_dimension_Node03
Node04 store_dimension_Node04
Node05 store_dimension_Node05
Administrator's Guide
ALL NODES automatically segments the projections across all five nodes in the cluster without
specifically referring to each node.
OFFSET ensures that the same data is not stored on the same nodes for each of the buddies.
The first buddy uses OFFSET 1 to shift the storage locations by 1 and the second buddy uses
OFFSET 2 to shift the storage locations by 1. This is critical to ensure K-safety.
CREATE PROJECTION retail_sales_fact_P1 (
C1_retail_sales_fact_store_key ENCODING RLE ,
C2_retail_sales_fact_pos_transaction_number ,
C3_retail_sales_fact_sales_dollar_amount ,
C4_retail_sales_fact_cost_dollar_amount )
AS SELECT T_retail_sales_fact.store_key,
T_retail_sales_fact.pos_transaction_number,
T_retail_sales_fact.sales_dollar_amount,
T_retail_sales_fact.cost_dollar_amount
FROM retail_sales_fact T_retail_sales_fact
ORDER BY T_retail_sales_fact.store_key
SEGMENTED BY HASH(T_retail_sales_fact.pos_transaction_number) ALL NODES;
----------------------------------------------------------- Projection #
: 6
-- Projection storage (KBytes) : 4.8e+06
-- Note: This is a super projection for table: retail_sales_fact
CREATE PROJECTION retail_sales_fact_P1_B1 (
C1_retail_sales_fact_store_key ENCODING RLE ,
C2_retail_sales_fact_pos_transaction_number ,
C3_retail_sales_fact_sales_dollar_amount ,
C4_retail_sales_fact_cost_dollar_amount )
AS SELECT T_retail_sales_fact.store_key,
T_retail_sales_fact.pos_transaction_number,
T_retail_sales_fact.sales_dollar_amount,
T_retail_sales_fact.cost_dollar_amount
FROM retail_sales_fact T_retail_sales_fact
ORDER BY T_retail_sales_fact.store_key
SEGMENTED BY HASH(T_retail_sales_fact.pos_transaction_number) ALL NODES OFFSET 1;
----------------------------------------------------------- Projection #
: 6
-- Projection storage (KBytes) : 4.8e+06
-- Note: This is a super projection for table: retail_sales_fact
CREATE PROJECTION retail_sales_fact_P1_B2 (
Administrator's Guide
Administrator's Guide
Segmented projections
P1 and its buddy, B1, are projections for the retail_sale_fact table. They were created using
the ALL NODES syntax, so HP Vertica automatically segments the projections across all three
nodes.
Unsegmented Projections
Currently three unsegmented superprojections exist for the store_dimension table, one for
each node, as follows:
Node
Unsegmented Projection
Node01 store_dimension_Node01
Node02 store_dimension_Node02
Node03 store_dimension_Node03
To support an additional node, replacement projections need to be created for the segmented
projections, P1 and B1. The new projections could be called P2 and B2, respectively. Additionally, an
unsegmented superprojection (store_dimension_Node04) needs to be created for the dimension
table on the new node (Node04).
Administrator's Guide
T_retail_sales_fact.pos_transaction_number,
T_retail_sales_fact.sales_dollar_amount,
T_retail_sales_fact.cost_dollar_amount
FROM retail_sales_fact T_retail_sales_fact
ORDER BY T_retail_sales_fact.store_key
SEGMENTED BY HASH(T_retail_sales_fact.pos_transaction_number) ALL NODES;
----------------------------------------------------------- Projection # : 6
-- Projection storage (KBytes) : 4.8e+06
-- Note: This is a super projection for table: retail_sales_fact
CREATE PROJECTION retail_sales_fact_P1_B1 (
C1_retail_sales_fact_store_key ENCODING RLE ,
C2_retail_sales_fact_pos_transaction_number ,
C3_retail_sales_fact_sales_dollar_amount ,
C4_retail_sales_fact_cost_dollar_amount )
AS SELECT T_retail_sales_fact.store_key,
T_retail_sales_fact.pos_transaction_number,
T_retail_sales_fact.sales_dollar_amount,
T_retail_sales_fact.cost_dollar_amount
FROM retail_sales_fact T_retail_sales_fact
ORDER BY T_retail_sales_fact.store_key
SEGMENTED BY HASH(T_retail_sales_fact.pos_transaction_number) ALL NODES
OFFSET 1;
----------------------------------------------------------
ALL NODES automatically places a complete copy of the superprojection on each of the three
original nodes.
CREATE PROJECTION store_dimension (
C0_store_dimension_floor_plan_type ENCODING RLE ,
C1_store_dimension_photo_processing_type ENCODING RLE ,
C2_store_dimension_store_key ,
C3_store_dimension_store_name ,
C4_store_dimension_store_number ,
C5_store_dimension_store_street_address ,
C6_store_dimension_store_city ,
C7_store_dimension_store_state ,
C8_store_dimension_store_region ,
C9_store_dimension_financial_service_type ,
C10_store_dimension_selling_square_footage ,
C11_store_dimension_total_square_footage ,
C12_store_dimension_first_open_date ,
C13_store_dimension_last_remodel_date )
Administrator's Guide
AS SELECT T_store_dimension.floor_plan_type,
T_store_dimension.photo_processing_type,
T_store_dimension.store_key,
T_store_dimension.store_name,
T_store_dimension.store_number,
T_store_dimension.store_street_address,
T_store_dimension.store_city,
T_store_dimension.store_state,
T_store_dimension.store_region,
T_store_dimension.financial_service_type,
T_store_dimension.selling_square_footage,
T_store_dimension.total_square_footage,
T_store_dimension.first_open_date,
T_store_dimension.last_remodel_date
FROM store_dimension T_store_dimension
ORDER BY T_store_dimension.floor_plan_type, T_store_dimension.photo_processing_type
UNSEGMENTED ALL NODES;
Administrator's Guide
Implementing Security
Implementing Security
In your HP Vertica database, there are three primary security operations:
l
Connection encryption prevents the interception of data and authenticates the identity of the
server and the client.
Client authorization (managing users and privileges) controls what users can access and
change in the database.
Client Authentication
To gain access to HP Vertica, a user or client application must supply the name of a valid user
account. You can configure HP Vertica to require just a user name,. However, you are likely to
require an additional means of authentication, such as a password.
You can use different authentication methods based on:
l
Connection type
User name for the client that is attempting to access the server
Connection Encryption
To secure the connection between the client and the server, you can configure HP Vertica and
database clients to use Secure Socket Layer (SSL) to communicate. HP Vertica uses SSL to:
l
Authenticate the server so the client can confirm the server's identity. HP Vertica supports
mutual authentication in which the server can also confirm the identity of the client. This
authentication helps prevent "man-in-the-middle" attacks.
Encrypt data sent between the client and database server to significantly reduce the likelihood
that the data can be read if the connection between the client and server is compromised.
Verify that data sent between the client and server has not been altered during transmission.
Administrator's Guide
Implementing Security
Client Authorization
Database users should have access to just the database resources they need to perform their
required tasks. For example, some users need to query only specific sets of data. To prevent
unauthorized access to additional data, you can limit their access to just the data that they need to
run their queries. Other users should be able to read the data but not be able to modify or insert new
data. Still other users might need more permissive access, including the right to create and modify
schemas, tables, and views, or grant other users access to database resources.
A collection of SQL statements control authorization for the resources users can access. For
details, see Managing Users and Privileges, in particular About Database Privileges.
You can also use roles to grant users access to a set of privileges, rather than directly grant the
privileges for each user. See About Database Roles.
Use the GRANT Statements to assign privileges to users and the REVOKE Statements to repeal
privileges.
Administrator's Guide
Implementing Security
Client Authentication
When a client (the user who runs a client application or the client application itself) connects to the
HP Vertica database server, it supplies the HP Vertica database user name to gain access. HP
Vertica restricts which database users can connect through client authentication, a process
where the database server establishes the identity of the requesting client and determines whether
that client is authorized to connect to the HP Vertica server using the supplied credentials.
HP Vertica offers several client authentication methods. You can configure HP Vertica to require
just a user name for connections, but you probably require more secure authentication, such as a
password at a minimum.
LOCALAuthenticating users or applications that are trying to connect from the same node on
which the database is running.
HOSTAuthenticating users or applications that are trying to connect from a node that has a
different IPv4or IPv6 address than the database.
For more information, see IPv4 and IPv6 for Client Authentication.
The DBADMINuser manages the client authentication information that the database uses to
authenticate users.
HP Vertica takes the following steps to authenticate users:
1. When a user or application attempts to connect to an HP Vertica database, the system checks
to see if the user is a DBADMINuser.
DBADMINusers can access the database at all times, unless their access is specifically
blocked by an authentication method, such as reject.
Administrator's Guide
Implementing Security
Note: For the DBADMIN user to be able to perform all Admintools functions, the
DBADMIN must always be able to authenticate by LOCAL TRUST or LOCAL
PASSWORD (the default for DBADMIN user). If you have changed DBADMIN user
authentication from LOCAL TRUST or LOCAL PASSWORD, use the
ALTERAUTHENTICATION statement to once again give the DBADMIN user LOCAL
TRUST or LOCAL PASSWORD authentication.
2. For non-DBADMINusers, the database checks to see if the user is associated with an
authentication method through a GRANTstatement. If a user has been associated with more
than one authentication method, HP Vertica tries to authenticate the user with the higher
priority authentication method.
Note: For detailed information about how authentication priorities work, see Priorities for
Client Authentication Methods.
If the user presents the correct credentials, the database allows the user to log in .
The DBADMINuser can grant an authentication method to users or user roles. The
DBADMINuser can also create a default authentication method that HP Vertica uses when no
authentication has been associated with a user or role.
3. If the user is not associated with an authentication method, the database checks to see if the
DBADMINhas established a default authentication method.
4. If the DBADMINhas specified a default authentication method, the database authenticates the
user using that default method.
5. If you have not specified a default authentication method, the database checks to see if the
DBADMIN user has defined any authentication methods. If the DBADMIN has not, no
authentication information exists in the database. However, if a password exists, the
DBADMINuser can log in.
6. If authentication information exists, HP Vertica rejects the user's request to connect to the
database. The DBADMINhas not granted an authentication method for that user nor has the
DBADMINdefined a default authentication method for all users ('public').
7. If authentication records exist in the database, HP Vertica uses implicit trust/implicit password
to authenticate the user.
Administrator's Guide
Implementing Security
If you are using a literal IPv6 address in a URL, you must enclose the IPv6 address in square
brackets as shown in the following examples:
=> ALTER
=> ALTER
=>ALTER
=> ALTER
=> ALTER
AUTHENTICATION
AUTHENTICATION
AUTHENTICATION
AUTHENTICATION
AUTHENTICATION
Ldap
Ldap
Ldap
Ldap
Ldap
SET
SET
SET
SET
SET
host='ldap://[1dfa:2bfa:3:45:5:6:7:877]';
host='ldap://[fdfb:dbfa:0:65::177]';
host='ldap://[fdfb::177]';
host='ldap://[::1]';
host='ldap://[1dfa:2bfa:3:45:5:6:7:877]:5678';
If you are working with a multi-node cluster, any IP/netmask settings in (HOST, HOST TLS,
HOSTNOTLS) must match all nodes in the cluster. This setup allows the database owner to
authenticate with and administer every node in the cluster. For example, specifying 10.10.0.8/30
allows a CIDR address range of 10.10.0.810.10.0.11.
For detailed information about IPv6 addresses, see RFC1924 and RFC2732.
Trust authenticationAuthorizes any user that connects to the server using a valid user name.
Reject authenticationBlocks the connection and prevents the requesting client from
accessing the database.
Administrator's Guide
Implementing Security
authentication service with single sign-on and trusted third-party certificate authority. GSS
authentication uses the GSS-APIstandard and provides compatibility with non-MITKerberos
implementations, such as those for Java and Windows clients.
Hash authenticationSends encrypted passwords hashed by the MD5 algorithm or the more
secure SHA-512 method over the network. The server provides the client with salt.
TLSauthenticationAuthenticates the client using digital certificates that contain a public key.
Transport Layer Security (TLS)is the successor to Secure SocketsLayer(SSL)authentication.
Host:Remote connection to the database from different hosts, each with their own IPv4or IPv6
address and host parameters. For more information, see IPv4 and IPv6 for Client
Authentication.
You can designate host authentication with or without TLS.
Local?
Host?
GSS
No
Yes
Administrator's Guide
Implementing Security
Authentication Method
Local?
Host?
Ident
Yes
No
LDAP
Yes
Yes
Hash
Yes
Yes
Reject
Yes
Yes
TLS
No
Yes
Trust
Yes
Yes
How To
Administrator's Guide
Implementing Security
Password
Trust
LDAP
GSS
Ident
Hash
Reject
TLS
authentication method.
LDAP
Ident
GSS
Administrator's Guide
Implementing Security
Associate an authentication
the database.
Administrator's Guide
Implementing Security
When you create authentication records using CREATE AUTHENTICATION,specify the following
information.
What you
need to
specify
Description
Access
attempting to connect:
l
'gss'
'ident'
'ldap'
'hash'
'reject'
'trust'
'tls'
LOCAL
HOST
HOSTNOTLS
HOSTTLS
method
Host
IPaddress
connect. This can be an IPv4 address or an IPv6 address. For more information,
see IPv4 and IPv6 for Client Authentication.
The following examples show how to create authentication records that are stored in the catalog.
When you create an authentication record using CREATEAUTHENTICATION, HP Vertica
automatically enables it.
This example creates an authentication method named localpwd to authenticate users who are
trying to log in from a local host using a password:
Administrator's Guide
Implementing Security
This example creates an authentication method named v_ldap that uses LDAP over TLSto
authenticate users logging in from the host with the IPv4 address 10.0.0.0/23:
=>CREATE AUTHENTICATION v_ldap METHOD 'ldap' HOST TLS '10.0.0.0/23';
This example creates an authentication method named v_kerberos to authenticate users that are
trying to connect from any host in the networks 2001:0db8:0001:12xx:
=>CREATE AUTHENTICATION v_kerberos METHOD 'gss' HOST '2001:db8:1::1200/56';
This example creates an authentication method named, RejectNoSSL, that rejects users from any
IPaddress that are trying to authenticate without SSL/TLS:
=>CREATE AUTHENTICATION RejectNoSSL METHOD 'reject' HOST NO TLS '0.0.0.0/0';
--IPv4
--IPv6
See Also
l
Administrator's Guide
Implementing Security
The most specific IPaddresses have priority over the least specific IPaddresses
For example, theIPv4 address 10.3.4.128/25 has priority over 10.3.0.0/24, which in turn has
priority over 10.3.0.0/16. The IPv6 address 2001:db8:ab::123/128 has priority over
2001:db8:1::1200/56.
Reject
Hash
Trust
If the highest priority authentication method is Ident and authentication fails, HP Vertica tries the
next highest priority authentication method, regardless of what method it uses.
If the next attempt does not use Ident authentication and fails, the authentication process ends.
However, if the next attempt uses Ident and fails, HP Vertica continues to the next highest
priority method. This process continues until authentication is successful or a non-Ident
authentication attempt fails.
Administrator's Guide
Implementing Security
If the highest priority method is LDAP and authentication fails, HP Vertica searches for the next
highest priority LDAP method. Authentication attempts continue until the authentication is
successful, or there are no additional LDAP authentication methods that satisfy the connection
criteria.
Note that if a user not found error occurs during LDAPauthentication, the retry connection
attempt initiates only if you set the ldap_continue parameter to yes.
For all other authentication types, HP Vertica tries the highest priority authentication method
associated with that user. If that authentication fails, the authentication process stops.
For example, suppose there are two client authentication methods associated with a user, as
follows:
=>
=>
=>
=>
=>
=>
When user tries to connect to the database, HP Vertica first tries auth_name2 to authenticate
because it has a higher priority. If that fails, HP Vertica tries auth_name1. If that fails,
authentication fails.
If you do not specify a priority, or omit the <priority_value> when using ALTER
AUTHENTICATION, HP Vertica sets the priority to 0.
Administrator's Guide
Implementing Security
high priority, such as 10,000. Grant this method to the DBADMINuser, and set the priority using
ALTERAUTHENTICATION.
With the high priority, this new authentication method supersedes any authentication methods you
create for PUBLIC (which includes the DBADMINuser). Even if you make changes to
PUBLICauthentication methods, the DBADMIN still has access.
Note: For the DBADMIN user to be able to perform all Admintools functions, the DBADMIN
must always be able to authenticate by LOCAL TRUST or LOCAL PASSWORD (the default
for DBADMIN user). If you have changed DBADMIN user authentication from LOCAL TRUST
or LOCAL PASSWORD, use the ALTERAUTHENTICATION statement to once again give
the DBADMIN user LOCAL TRUST or LOCAL PASSWORD authentication.
See Also
l
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
=>REVOKEAUTHENTICATIONv_gss FROMDBprogrammer;
Change a Parameter
Set the system_users parameter for ident1 authentication to root:
=> CREATEAUTHENTICATION ident1 METHOD 'ident' LOCAL;
=> ALTERAUTHENTICATION ident1 SET system_users='root';
Administrator's Guide
Implementing Security
Change the IPaddress and specify the parameters for an LDAP authentication method named
Ldap1.
In this example, you specify the bind parameters for the LDAPserver. HP Vertica connects to the
LDAP server, which authenticates the HP Vertica client. If the authentication succeeds, HP
Vertica authenticates any users who have been granted the Ldap1 authentication method on the
designated LDAPserver:
=> CREATE AUTHENTICATION Ldap1 METHOD 'ldap' HOST '172.16.65.196';
=>ALTER AUTHENTICATION Ldap1 SET host='ldap://172.16.65.177',
binddn_prefix='cn=', binddn_suffix=',dc=qa_domain,dc=com';
Change the IPaddress, and specify the parameters for an LDAP authentication method named
Ldap1. Assume that HP Vertica does not have enough information to create the distinguished name
(DN)for a user attempting to authenticate. Therefore, in this case, you must specify to use LDAP
bind and search:
=> CREATE AUTHENTICATION LDAP1 METHOD 'ldap' HOST '172.16.65.196';
=> ALTER AUTHENTICATION Ldap1 SET host='ldap://172.16.65.177',
basedn='dc=qa_domain,dc=com',binddn='cn=Manager,dc=qa_domain,
dc=com',search_attribute='cn',bind_password='secret';
ALTERAUTHENTICATIONvalidates the parameters you enter. If there are errors, it disables the
authentication method that you are trying to modify.
See Also
l
Administrator's Guide
Implementing Security
To delete an authentication record for a method that has been granted to a user, use the
CASCADEkeyword:
=> CREATEAUTHENTICATIONlocalpwd METHOD 'password' LOCAL;
=>GRANTAUTHENTICATIONlocalpwd TOjsmith;
=> DROPAUTHENTICATION localpwd CASCADE;
See Also
l
CLIENT_AUTH
CLIENT_AUTH_PARAMS
PASSWORD_AUDITOR
USER_CLIENT_AUTH
To determine the details behind the client authentication used for a particular user session, query
the following tables in the V_MONITORschema:
Administrator's Guide
Implementing Security
SESSIONS
USER_SESSIONS
Administrator's Guide
Implementing Security
Password Authentication
One of the simplest ways to authenticate a client connection to the database is to assign the user
account an HP Vertica password. If a user account has a password, the user or client for that
account must supply the correct password to connect to the database. However, if the user
account does not have a password, and HP Vertica is not configured to use another form of client
authentication, the user account is always allowed to log in.
HP Vertica stores passwords in an encrypted format to prevent potential theft. However, the
transmission of the password to HP Vertica is in plain text. Thus, it is possible for a "man-in-themiddle" attack to intercept the password.
To secure the login, Hewlett-Packard recommends implementing SSL security or hash
authentication.
Administrator's Guide
Implementing Security
Profiles
You set password policies using profiles. A profile is a group of parameters that includes
requirements for user passwords.
A profile controls:
l
How many times users must change their passwords before they can reuse an old password.
How many times a user can fail to log in before the account is locked.
Minimum number of capital letters, lowercase letters, digits, and symbols required in a
password.
To set a user's password policy, assign the user to a profile. To enforce different password policies
for different users, create multiple profiles. For example, you might create one profile for interactive
users, requiring them to frequently change their passwords. You might create another profile for
user accounts that are not required to change passwords.
You do not explicitly assign users a profile when you create them.
You can change the policy parameters in the DEFAULT profile, but you cannot delete it.
Important: During upgrades from versions of HP Vertica earlier than version 5.0, each
database receives a DEFAULT profile. All users are then automatically assigned to that
Administrator's Guide
Implementing Security
profile.
The profiles that you create can inherit some or all of their policy parameters from the DEFAULT
profile. When you create a profile using CREATE PROFILE, a parameter inherits its value from the
DEFAULT profile if:
l
If you change a parameter in the DEFAULT profile, you also change that parameter's value in every
profile that inherits the parameter from DEFAULT.
Changes to a profile's policies for password content do not have an immediate effect on the users.
When HP Vertica does not test user's passwords to verify that they comply with the new password
criteria. Instead, the changed settings only affect the users the next time they change their
password. To make sure that users comply with the new password policy, use the ALTER USER
statement to expire user passwords. HP Vertica prompts users with expired passwords to change
their passwords when they next log in.
Note: Only the profile settings for how many failed login attempts trigger Account Locking and
how long accounts are locked have an effect on password authentication methods such as
LDAP or GSS. All password complexity, reuse, and lifetime settings affect only passwords
that HP Vertica manages.
See Also
l
PROFILES
Password Expiration
User profiles control how often users must change their passwords. Initially, the DEFAULT profile
is set so that passwords never expire.
Important: Password expiration has no effect on any of the user's current sessions.
Administrator's Guide
Implementing Security
When a password expires, the user must change the password on the next login. However, you can
set a PASSWORD_GRACE_TIME in any individual user's profile, allowing that user to log in after
the expiration. After the password expires, HP Vertica issues a warning about the password
expiration but continues to recognize the password.
After the grace period ends, users must change their passwords to log in, unless they have
changed them already in response to the warning.
Expiring a Password
You can expire a user's password immediately using the ALTER USER statement's PASSWORD
EXPIRE parameter. By expiring a password, you can:
l
Account Locking
In a profile, you can set a password policy for how many consecutive failed login attempts a user
account is allowed before locking.This locking mechanism helps prevent dictionary-style bruteforce attempts to guess users' passwords.
Set this value using the FAILED_LOGIN_ATTEMPTS parameter using the CREATE PROFILE or
ALTER PROFILE statement.
HP Vertica locks any user account that has more consecutive failed login attempts than the value
to which you set FAILED_LOGIN_ATTEMPTS. The user cannot log in to a locked account, even by
supplying the correct password.
ManuallyIf you are a superuser, you can manually unlock the account using the ALTER
USER command.
Note: A superuser account cannot be locked, because it is the only user that can unlock
accounts. For this reason, choose a very secure password for a superuser account. See
Password Guidelines for suggestions.
Administrator's Guide
Implementing Security
Password Lock Time SettingSpecify the number of days until an account unlocks in the
PASSWORD_LOCK_TIME parameter of the user's profile. HP Vertica automatically unlocks the
account after the specified number of days has passed. If you set this parameter to UNLIMITED,
the user's account is never automatically unlocked, and a superuser must manually unlock it.
Password Guidelines
For passwords to be effective, they must be hard to guess. You need to protect passwords from:
l
Users who have knowledge of the password holder (family names, dates of birth, etc.)
Use Profiles to enforce good password practices (password length and required content). Make
sure database users know the password guidelines, and encourage them not to use personal
information in their passwords.
Strong Passwords
Use the following password guidelines, published by the Internet Engineering Task Force (IETF),
when you create passwords:
l
Use a password that is easy to remember, so you do not need to write it down. For example, use
i3atSandw1ches! instead of !a#^*!$&D)z.
Use a password that you can type quickly without having to look at the keyboard.
Weak Passwords
Avoid using the following practices to create a password:
l
Do not use your login or user name in any form (as-is, reversed, capitalized, doubled, and so on).
Do not use your spouse's, partner's, child's, parent's, friend's, or pet's name in any form.
Administrator's Guide
Implementing Security
Do not use other information easily obtained about you, including your date of birth, license plate
number, telephone number, Social Security number, make of your automobile, house address,
and so on.
Do not use a word contained in English or foreign language dictionaries, spelling lists, acronym
or abbreviation lists, or other lists of words.
See Also
l
LDAPParameters
Configuring MultipleLDAPServers
Administrator's Guide
Implementing Security
LDAPAuthentication Definitions
DBADMINAuthentication Access
LDAPParameters
LDAPAnonymous Binding
IPaddress and host name for the LDAP server. HP Vertica supports IPv4 and IPv6 addresses.
open-ldap-tools package installed on at least one node. This package includes ldapsearch.
LDAPAuthentication Definitions
The following definitions are important to remember for LDAPauthentication:
Parameter name
Description
Host
Common name
(CN)
Administrator's Guide
Implementing Security
Parameter name
Description
Domain
component (DC)
Distinguished
name (DN)
DC= com".
Organizational unit
Unit in the organization with which the user is associated, for example,
(OU)
Vertica Users.
sAMAccountName An Active Directory user account field. This value is usually the attribute to
be searched when you use bind and search against the Microsoft Active
Directory server.
UID
Bind
Service account
An LDAP user account that can be used to log in to the LDAP server during
bind and search. This account's password is usually shared.
Anonymous
Allows a client to connect and search the directory (bind and search)
binding
ldapsearch
basedn
binddn
search_attribute
Text to search for to locate the user record. The default is UID.
DBADMINAuthentication Access
The DBADMINuser must have access to the database at all times.
The DBADMINaccount must authenticate against the database using local trust or local hash
authentication.
Administrator's Guide
Implementing Security
The next example shows how you configure local hash authentication for DBADMIN.They allow
the user to access the HP Vertica database using the assigned password from any IPv4 address.
The DBADMINuser can access the database using vsql -h, the Administration Tools, or any
other tools that connects to HP Vertica:
=>CREATEAUTHENTICATIONv_dbadmin_hash METHOD'hash' HOST '0.0.0.0/0';
=> GRANTAUTHENTICATIONv_dbadmin_hash TOdbadmin;
=> ALTERAUTHENTICATIONv_dbadmin_hash PRIORITY10000;
=> SELECTSET_CONFIG_PARAMETER('SecurityAlgorithm', 'SHA512');
Note: HP Vertica supports IPv4 and IPv6 addresses. For more information, see IPv4 and IPv6
for Client Authentication.
LDAPParameters
There are several parameters that you need to configure for LDAPauthentication.
General LDAPParameters
There are four parameters you can use when configuring for either LDAPbind or LDAPbind and
search:
Administrator's Guide
Implementing Security
Parameter name
Description
host
starttls
basedn
ldap_continue
LDAPBind Parameters
Use the following parameters when authenticating with LDAPbind to create the bind name string:
Administrator's Guide
Implementing Security
Paramete
r name
Description
binddn_
prefix
binddn_
suffix
domain_
prefix
email_
suffix
To create the bind name string, you must provide one of the following,
l
domain_name
email_suffix
Otherwise, HP Vertica performs a bind and search operation instead of a bind operation.
Administrator's Guide
Implementing Security
Parameter name
Description
binddn
bind_password
search_attribute
The following example shows how to set these three attributes. In this example, it sets
l
binddn to cn=Manager,dc=qa_domain,dc=com
bind_password to secret
search_attribute to cn
=> ALTER AUTHENTICATION auth_method_name SET host=:'ldap://engvmqa13',
basedn='dc=qa_domain,dc=com',binddn='cn=Manager,dc=qa_domain,dc=com',
bind_password='secret',search_attribute='cn';
The binddn and bind_password parameters are optional. If you omit them, HP Vertica performs an
anonymous search.
BindUse LDAPbind when HP Vertica connects to the LDAP server and binds using the CN
and password. (These values are the username and password of the user logging into the
database). Use the bind method when your LDAP account's CN field matches that of the
username defined in your database.
Bind and searchUse LDAPbind and search when your LDAP account's CN field is a user's
full name or does not match the username defined in your database. For bind and search, the
username is usually in another field such as UID or sAMAccountName in a standard Active
Directory environment. Bind and search requires your organization's Active Directory
information. This information allows HP Vertica to log into the LDAP server and search for the
specified field.
Administrator's Guide
Implementing Security
If you are using bind and search, having a service account simplifies your server side
configuration. In addition, you do not need to store your Active Directory password.
For details, see
l
LDAPAnonymous Binding
Anonymous binding is an LDAP server function. Anonymous binding allows a client to connect and
search the directory (bind and search) without logging in. You do not need to include binddn and
bindpasswd.
You also do not need to log in when you configure LDAP authentication using Management
Console.
If the LDAP server does not support SSL on that port, authentication fails.
If you specify a host URLthat starts with ldap and set the LDAPstarttls parameter, the HP
Vertica server sends a StartTLS request. This request determines if the LDAP server supports TLS
on the specified port or on the default LDAP port (389).
=> ALTER AUTHENTICATION Ldap1 SET host='ldaps://abc.dc.com', binddn_prefix='CN=',
binddn_suffix=',OU=Unit2,DC=dc,DC=com', basedn='dc=DC,dc=com',
tls_cacert='/home/dc.com.ca.cer', starttls='hard', tls_reqcert='never';
If the LDAP server does not support TLS on that port, the result depends on the value of the
starttls parameter:
Administrator's Guide
Implementing Security
starttls = soft:The HP Vertica server proceeds with the authentication but does not use
TLS.
Description
TLS_REQCERT
TLS_CADIR
TLS_CACERT
If you do not provide one or more of these parameters, the LDAP server checks to see if the
LDAPNOINIT environment variable points to the ldap.conf file. If it does, the server uses the
parameters specified in the ldap.conf file. If the LDAP server cannot find the ldap.conf file,
authentication fails.
The following example shows how to specify the TLS parameters and the LDAP parameters when
configuring LDAP over SSL/TLS:
=>CREATE AUTHENTICATION LDAP1 METHOD 'ldap' HOST :clientIP = '172.16.65.177';
GRANTAUTHENTICATIONldap1 TO user1;
ALTER AUTHENTICATION Ldap1 SET host='ldaps://abc.dc.com', binddn_prefix='CN=',
binddn_suffix=',OU=Unit2,DC=dc,DC=com', basedn='dc=DC,dc=com',
tls_cacert='/home/dc.com.ca.cer', starttls='hard', tls_reqcert='never';
Administrator's Guide
Implementing Security
Create a service account with your LDAPserver. A service account is a single account that
is specifically set up so that users in a given organization can share an account that configures
LDAPaccess. Create a service account and use that in your LDAP URL to avoid use of account
names and passwords, which change often. If you add, remove, or change users, you do not
have to modify the LDAPURL. Having a service account allows you to restrict individual users
from searching the LDAPserver, but it allows applications like HP Vertica to search the server.
Set up an organizational unit (OU). Create an Active Directory OU, which is a group of users
in a given organization. Add all the HP Vertica users to theOU, and specify theOUin the
LDAPURL. Doing so allows the LDAP server to search just the HP Vertica OU for the user,
minimizing search time. In addition, using OUs prevents changes to the users' OUs for other
applications.
Administrator's Guide
Implementing Security
user_name | client_authentication_name
-----------+---------------------------r1
| ldap
(1 row)
3. Run ldapsearch from an HP Vertica node against your LDAP or AD server. Verify the
connection to the server and identify the values of relevant fields. Running ldapsearch helps
you build the client authentication string needed to configure LDAP authentication.
In the following example, ldapsearch returns the CN, DN, and sAMAccountName fields (if
they exist) for any user whose CN contains the username jsmith. This search succeeds only
for LDAPservers that allow anonymous binding:
ldapsearch -x -h 10.10.10.10 -b "ou=Vertica Users,dc=CompanyCorp,dc=com"
'(cn=jsmith*)' cn dn uid sAMAccountName
ldapsearch returns the following results. The relevant information for LDAPbind is in bold:
# extended LDIF
#
# LDAPv3
# base <ou=Vertica Users,dc=CompanyCorp,dc=com> with scope subtree
# filter: (cn=jsmith*)
# requesting: cn dn uid sAMAccountName
#
# jsmith, Users, CompanyCorp.com
dn:cn=jsmith,ou=Vertica Users,dc=CompanyCorp,dc=com
cn: jsmith
uid: jsmith
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
4. Create a new authentication record based on the information from ldapsearch. In the
ldapsearch entry, the CN is username jsmith, so you do not need to set it. HP Vertica
automatically sets the CN to the username of the user who is trying to connect. HP Vertica
uses that CN to bind against the LDAP server.
=> CREATEAUTHENTICATIONv_ldap_bind METHOD 'ldap' HOST'0.0.0.0/0';
GRANTAUTHENTICATIONv_ldap_bind TOpublic;
ALTERAUTHENTICATIONv_ldap_bind SET
host='ldap://10.10.10.10/',
basedn='DC=CompanyCorp,DC=com',
binddn_prefix='cn=',
Administrator's Guide
Implementing Security
binddn_suffix='OU=VerticaUsers,DC=CompanyCorp,DC=com';
3. Review the results that ldapsearch returns.The relevant information for bind and search is in
bold:
# extended LDIF
#
# LDAPv3
# base <OU=Vertica Users,DC=CompanyCorp,DC=com> with scope subtree
# filter: (cn=John*)
# requesting: cn dn sAMAccountName
#
# John Smith, Vertica Users, CompanyCorp.com
dn: CN=John Smith,OU=Vertica Users,DC=CompanyCorp,DC=com
cn: John Smith
sAMAccountName: jsmith
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
4. Create the client authentication record. The sAMAccountName attribute contains the
username you wantjsmith. Set your search attribute to that field so that the search finds the
appropriate account.
Administrator's Guide
Implementing Security
Configuring MultipleLDAPServers
If you need to configure multiple LDAPservers that have different URLs, create a separate
authentication record for each server. Use the PRIORITYkeyword to indicate which search the
LDAP server performs first.
The following statements create two authentication methods, vldap1 and vldap2. They specify
that the LDAP server first search the entire directory (basedn=dc=example,dc=com) for a DN with
an OU attribute Sales. If the first search returns no results, or otherwise fails, the LDAP server next
searches for a DN with the OU attribute Marketing:
=> CREATEAUTHENTICATION vldap1 method "ldap" HOST10.0.0.0/8;
ALTERAUTHENTICATIONvldap1 SET
host='ldap://ldap.example.com/search',
basedn='dc=example,dc=com',
search_attribute='Sales'
PRIORITY1;
GRANTAUTHENTICATIONvldap1 to public;
=>CREATEAUTHENTICATIONvldap2 method "ldap"HOST10.0.0.0/8;
ALTERAUTHENTICATIONvldap2 SET
host='ldap://ldap.example.com/search',
basedn='dc=example,dc=com',
search_attribute='Marketing'
PRIORITY0;
GRANTAUTHENTICATIONvldap1 to public;
Administrator's Guide
Implementing Security
Topics in this section describe how to configure the HP Vertica server and clients for Kerberos
authentication. This section does not describe how to install, configure, or administer a Key
Distribution Center.
To install the Kerberos 5 GSS-API distribution for your operating system, see the MIT Kerberos
Distribution Page.
Kerberos Requirements
You must meet the following minimum requirements to use Kerberos authentication with the HP
Vertica server and client drivers.
Kerberos Server
Your network administrator should have already installed and configured one or more Kerberos Key
Distribution Centers (KDC). The KDC must be accessible from every node in your Vertica Analytic
Database cluster.
The KDC must support Kerberos 5 using GSS-API. For details, see the MIT Kerberos Distribution
Page.
Client Package
The Kerberos 5 client package contains software that communicates with the KDC server. This
package is not included as part of the HP Vertica Analytics Platform installation. Kerberos software
is built into Microsoft Windows. If you are using another operating system, you must obtain and
install the client package.
If you do not already have the Kerberos 5 client package on your system, download it. Then, install
the package on each HP Vertica server and each HP Vertica client used in Kerberos authentication,
except the KDC itself.
Refer to the Kerberos documentation for installation instructions, found at:
l
Client/Server Identity
Configure as Kerberos principals:
Administrator's Guide
Implementing Security
Restartthe Database
Administrator's Guide
Implementing Security
Kerberized HP Vertica clients request access to this service when they authenticate to the
database.
System processes like the Tuple Mover use this identity when they authenticate to external
services such as Hadoop.
Principals
A principal has three elements: a name, a host name, and a realm. You choose the name and realm,
but the host name must match the value that is supplied by the operating system. Typically this is
the fully-qualified host name. If the host name part of your principal doesn't match the value
supplied by the operating system, Kerberos authentication fails.
Some systems use a hosts file (/etc/hosts) to define host names. A hosts file can define more than
one name for a host. The operating system supplies the first entry, so use that in your principal. For
example, if your hosts file contains:
192.168.1.101 node1.example.com node1
Keytab Files
Principals are stored in encrypted keytab files. The keytab file contains the credentials for the HP
Vertica principal. The keytab allows the HP Vertica server to authenticate itself to the KDC. You
need the keytab so that Vertica Analytic Database does not have to prompt for a password.
Create one principal for each node in your cluster. You can then either create individual keytab files
(one for each node containing only that node's principal) or create one keytab file containing all the
principals.
l
Create one keytab file with all principals to simplify setup: all nodes have the same file,
making initial setup easier. If you add nodes later you either update (and redistribute) the global
keytab file or make separate keytabs for the new nodes. If a principal is compromised it is
compromised on all nodes where it is present in a keytab file.
Create separate keytab files on each node to simplify maintenance. Initial setup is more
involved as you must create a different file on each node, but no principals are shared across
nodes. If you add nodes later you create keytabs on the new nodes. Each node's keytab
contains only one principal, the one to use for that node.
Administrator's Guide
Implementing Security
Use kadmin if you are accessing the KDC on a remote server. If you have access to the
Kerberos administrator password, you can use kadmin on any machine where the Kerberos
5 client package is installed. When you start kadmin, the utility prompts you for the Kerberos
administrator's password. You might need root privileges on the client to run kadmin.
Repeat the ktadd command once per principal, either creating separate keytabs for each one
or adding them all to a single keytab file (such as krb5.keytab). If you are using a single file, see
the documentation for the -glob option in the MITKerberos documentation.
4. Copy each keytab file to the /etc folder on the corresponding cluster node. Use the same path
and file name on all nodes.
5. On each node, make the keytab file readable by the file owner who is running the database
process (typically, the Linux dbadmin user). For example, you can change ownership of the
Administrator's Guide
Implementing Security
Important: In a production environment, you must control who can access the keytab file
to prevent unauthorized users from delegating your server. For more information about
delegation (also known as impersonation), see Technet.Microsoft.com.
After you create a keytab file, you can use the klist command to view keys stored in the file:
$sudo /usr/kerberos/bin/klist -ke -t
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- ------------------------------------------------4 vertica/[email protected]
4 vertica/[email protected]
Administrator's Guide
Implementing Security
7. Use the ktpass command on the Windows server to configure the server principal name for the
host or service in Active Directory. Then, generate a .keytab file, as in the following example:
ktpass -out ./host.node1.example.com.keytab
-princ host/[email protected]
-mapuser node1
-mapop set -pass <password>
-crypto <encryption_type> -ptype <principal_type>
ktpass -out ./vertica.node1.example.com.keytab
-princ vertica/[email protected]
-mapuser node1
-mapop set -pass <password>
-crypto <encryption_type> -ptype <principal_type>
Administrator's Guide
Implementing Security
See the Kerberos documentation for information about other sections in this configuration file.
You must update the /etc/krb5.conf file to reflect your site's Kerberos configuration. The
simplest way to enforce consistency among all clients and servers in the Kerberos realm is to copy
the /etc/krb5.conf file from the KDC. Then, place this file in the /etc directory on each HP
Vertica cluster node.
The keytab file must be in the same location (/etc/krb5.keytab in this example) on all nodes.
3. Set the service name for the HP Vertica principal; for example, vertica:
=> ALTERDATABASE mydb SET KerberosServiceName = 'vertica';
Administrator's Guide
Implementing Security
Restartthe Database
For all settings to take effect, you must restart the database.
To request a ticket from the KDC server, call the kinit utility.
$kinit [email protected]
Password for [email protected]:
For each client, the GSS libraries require the following format for the HP Vertica service principal:
Kerberos_Service_Name@Kerberos_Host_Name
You can omit the realm portion of the principal because GSS libraries use the realm name of the
configured default (Kerberos_Realm) realm.
For information about client connection strings, see the following topics in the Connecting to HP
Vertica Guide:
l
Administrator's Guide
Implementing Security
Note: A few scenarios exist in which theHP Vertica server principal name might not match the
host name in the connection string. See Troubleshooting Kerberos Authentication for more
information.
In This Section
l
Configure ODBC and vsql Clients on Linux, HP-UX, AIX, MAC OSX, and Solaris
Configure ODBC and vsql Clients on Linux, HP-UX, AIX, MAC OSX,
and Solaris
This topic describes the requirements for configuring an ODBC or vsql client on Linux, HP-UX, AIX,
MAC OSX, or Solaris.
Domain
DNS lookup,
Administrator's Guide
Implementing Security
Ticket lifetime
Administrator's Guide
Implementing Security
If the vsql client is on the same machine you are connecting to, vsql connects through a UNIX
domain socket. This connection bypasses Kerberos authentication. When you authenticate with
Kerberos, especially if the client authentication method is configured as 'local', you must include
the -h hostname option. See Command Line Options in the Connecting to HP Vertica Guide.
1. On the vsql client, call the kinit utility:
$kinit [email protected]
Password for [email protected]:
2. Connect to HP Vertica, and provide the host and user principals in the connection string:
$./vsql -K vcluster.example.com -h myserver.example.com -U kuser
Welcome to vsql, the Vertica Analytic Database
interactive terminal.
Type: \h or \? for help with vsql commands
\g or terminate with semicolon to execute query
\q to quit
In the future, when you log in to vsql as kuser, vsql uses your cached ticket without prompting you
for a password.
You can verify the authentication method by querying the SESSIONS system table:
kuser=> SELECT authentication_method FROM sessions;
authentication_method
----------------------GSS-Kerberos
(1 row)
See Also
l
Administrator's Guide
Implementing Security
You can choose between two different setup scenarios for Kerberos authentication on ODBCand
vsql clients on Windows and ADO.NET:
l
Windows KDC on Active Directory with Windows Built-in Kerberos Client and HP Vertica
Windows KDC on Active Directory with Windows Built-in Kerberos Client and
HP Vertica
Kerberos authentication on Windows is commonly used with Active Directory, Microsoft's
enterprise directory service/Kerberos implementation.Typically your organization's network or IT
administrator performs the setup.
Windows clients have Kerberos authentication built into the authentication process. You do not
need any additional software.
Your login credentials authenticate you to the Kerberos server (KDC) when you:
l
Use a Windows instance that has been configured to use Kerberos through Active Directory
Administrator's Guide
Implementing Security
To set up Windows clients with Active Directory, refer to Step-by-Step Guide to Kerberos 5
(krb5 1.0) Interoperability.
To set up Windows clients with the ksetup utility, refer to the Ksetup page.
Administrator's Guide
Implementing Security
See Also
l
You can find documentation for the LoginModules in the com.sun.security.auth package, and
on the Krb5LoginModule web page.
Administrator's Guide
Implementing Security
Important: Carefully construct the JAAS login configuration file. If syntax is incorrect,
authentication fails.
You can configure JAAS-related settings in the java.security master security properties file. This
file resides in the lib/security directory of the JRE. For more information, see Appendix A in the
JavaTM Authentication and Authorization Service (JAAS) Reference Guide.
Create a JDBCLogin Context
The following example shows how to create a login context for Kerberos authentication on a
JDBCclient. The client uses the default JAASConfigName of verticajdbc and specifies that:
l
The user will not be prompted for a password if credentials cannot be obtained from the cache,
keytab file, or through a shared state.
verticajdbc {
com.sun.security.auth.module.Krb5LoginModule
required
useTicketCache=true
doNotPrompt=true;
};
Administrator's Guide
Implementing Security
If you prefer to use a password instead of calling the kinit utility, see the next section.
2. Connect to HP Vertica:
Properties props = new Properties();
props.setProperty("user", "kuser");
props.setProperty("KerberosServiceName", "vertica");
props.setProperty("KerberosHostName", "vcluster.example.com");
props.setProperty("JAASConfigName", "verticajdbc");
Connection conn = DriverManager.getConnection
"jdbc:vertica://myserver.example.com:5433/VMart", props);
The preceding example demonstrates the flexibility of JAAS. The driver no longer looks for a
cached ticket, and you do not have to call kinit. Instead, the driver takes the password and user
name and calls kinit on your behalf.
See Also
l
JavaTM Authentication and Authorization Service (JAAS) Reference Guide (external website)
Administrator's Guide
Implementing Security
Synchronize system clocks on all machines that participate in the Kerberos realm within a few
minutes of the KDC and each other
Clock skew can be problematic on Linux virtual machines that need to sync with the Windows Time
Service. Try the following to keep time in sync:
1. Using any text editor, open /etc/ntp.conf.
2. Under the Undisciplined Local Clock section, add the IP address for the Vertica Analytic
Database server. Then, remove existing server entries.
3. Log in to the server as root, and set up a cron job to sync time with the added IP address every
half hour, or as often as needed. For example:
# 0 */2 * * * /etc/init.d/ntpd restart
Administrator's Guide
Implementing Security
For more information, see Set Up Time Synchronization in the Installation Guide and the Network
Time Protocol website.
Administrator's Guide
Implementing Security
Backup/restore and the admin tools use the value of KerberosHostname, if it is set, in the Kerberos
principal used to authenticate. The same value is used on all nodes. If you have defined one
Kerberos principal per node, as recommended, this value does not match. To correct this, unset the
KerberosHostname parameter:
=> ALTERDATABASE mydb CLEARKerberosHostname;
Tip: On vsql clients, you set the host name portion of the server's principal name using the
-K KRB HOST command-line option. The default value is specified by the -h switch, which
is the host name of the machine on which the HP Vertica server is running. -K is
equivalent to the drivers' KerberosHostName connection string value.
Administrator's Guide
Implementing Security
Because the this connection string includes no explicit KerberosHostName parameter, the driver
defaults to the host in the URL (node01.example.com). If you overwrite the server-side
KerberosHostName parameter as abc, the client generates an incorrect principal.
To resolve this issue, explicitly set the clients KerberosHostName to the connection string, as in
this example:
jdbc:vertica://node01.example.com/vmart?user=kuser&kerberoshostname=abc
Connection load balancing is enabled, but the node against which the client authenticates might
not be the node in the connection string.
In this situation, consider changing all nodes to use the same KerberosHostName setting. When
you use the default to the host that was originally specified in the connection string, load
balancing cannot interfere with Kerberos authentication.
You have a DNS name that does not match the Kerberos host name.
For example, imagine a cluster of six servers, where you want hr-servers and financeservers to connect to different nodes on the Vertica Analytic Database cluster. Kerberos
authentication, however, occurs on a single (the same) KDC. In the following example, the
Kerberos service host name of the servers is server.example.com.
Suppose you have the following list of example servers:
server1.example.com
server2.example.com
server3.example.com
server4.example.com
192.16.10.11
192.16.10.12
192.16.10.13
192.16.10.14
Administrator's Guide
Implementing Security
server5.example.com 192.16.10.15
server6.example.com 192.16.10.16
For example:
$vsql -h finance-servers.example.com -K server.example.com
You do not have DNS set up on the client machine, so you must connect by IP only.
To resolve this issue, specify:
n
For example:
$vsql -h 192.168.1.12 -K server.example.com
There is a load balancer involved (Virtual IP), but there is no DNSname for the VIP.
Specify:
n
For example:
$vsql -h <virtual IP> -K server.example.com
Administrator's Guide
Implementing Security
You connect to HP Vertica using an IP address, but there is no host name to construct the
Kerberos principal name.
Provide the instance or host name for the HP Vertica as described in Inform HPVertica About
the Kerberos Principals and Keytab
You set the server-side KerberosHostName configuration parameter to a name other than the HP
Vertica node's host name, but the client cannot determine the host name based on the host
name in the connection string alone.
Rename the KerberosHostName to match the name of the HP Vertica node's host name. For
more information, see the following topics in the Connecting to HP Vertica Guide:
n
Administrator's Guide
Implementing Security
'NONE'
'MD5'
'SHA512'
The user-level parameter usually has precedence over the system-level parameter. However, if the
user-level parameter is 'NONE', HP Vertica hashes passwords with the algorithm assigned to the
system-level parameter value. If both parameters are 'NONE', HP Vertica uses the MD5 algorithm.
These values, which are stored in the PASSWORD_AUDITORsystem table, affect the security
algorithm that is actually used for hash authentication.
User-Level Parameter
Value
System-Level Parameter
Value
'NONE'
'NONE'
MD5
Administrator's Guide
Implementing Security
User-Level Parameter
Value
System-Level Parameter
Value
'NONE'
'MD5'
MD5
'NONE'
'SHA512'
SHA-512
'MD5'
'NONE'
MD5
'MD5'
'MD5'
MD5
'MD5'
'SHA512'
MD5
'SHA512'
'NONE'
SHA-512
'SHA512'
'MD5'
SHA-512
'SHA512'
'SHA512'
SHA-512
Administrator's Guide
Implementing Security
password changes.
You can use any of three approaches to changing the user password:
l
Manually set the user's user-level security algorithm to 'SHA512'. Then, change the users
password, as in the following statement:
=> ALTER USER username SECURITY_ALGORITHM 'SHA512' IDENTIFIED BY 'newpassword';
Set the user's password to expire immediately as in the following statement. After the password
expires, the user responds by changing it.
=>ALTERUSERusername PASSWORDEXPIRE;
All new passwords inherit the system-level security algorithm, which is SHA-512.
Administrator's Guide
Implementing Security
If users are trying to connect from an IPv6 address, the statement might look like this example:
=>CREATE AUTHENTICATION v_hash METHOD 'hash' HOST '2001:db8:ab::123/128';
2. Decide which password-hashing algorithm you want to use: MD5 or the more secure SHA-512.
3. Specify the security algorithm as follows:
n
At the system level, set the SecurityAlgorithm configuration parameter. This setting applies
to all users, unless their user-level security is set to another value:
=>ALTERDATABASE mydb SecurityAlgorithm = 'MD5';
=>ALTERDATABASE mydb SecurityAlgorithm = 'SHA512';
If you want users to inherit the system-level security, set their passwords to expire
immediately. Users must change their passwords before they log in again. Alternatively, you
can ask users to change their passwords. HP Vertica hashes all new passwords using the
system-level security algorithm.
n
At the user level, use ALTER USERto set the Security_Algorithm user parameter. Changing
this parameter at the user level overrides the system-level value:
=>ALTERUSERusername SECURITY_ALGORITHM 'MD5' IDENTIFIEDBY'newpassword';
=> ALTERUSER username SECURITY_ALGORITHM 'SHA512' IDENTIFIEDBY'newpassword';
4. Associate the v_hash authentication method with the desired users or user roles, using a
GRANTstatement:
=> GRANTAUTHENTICATIONv_hash to user1, user2, ...;
For more information about how these parameters work, see Hash Authentication Parameters.
Administrator's Guide
Implementing Security
the time it became open source. The introduction of TLS began with version 1, which is essentially
equal to SSL 3. You use openssl commands to create certificates and keys and TLS syntax to
create an authentication method.
Authentication methods define how clients connect to an HP Vertica server. Before you define a
TLS authentication method, you should understand what type of authentication methods your HP
Vertica server supports. You should also perform any prerequisite tasks.
In regards to SSL, your server can operate with:
l
No SSL
SSL Server Mode The client does not need certificate or key files.
SSL Mutual Mode The client needs certificate, key, and certificate authority files.
SSL modes are independent of authentication, except that the SSL client self-authentication
method requires that your server be set-up in SSL Mutual Mode. Otherwise, if you are not
implementing client self-authentication method, you can use TLS authentication with either SSL
Server Mode or SSL Mutual Mode.
Before you create a TLS authentication method, perform the pre-requisite tasks necessary for your
specific environment (for example, certificate creation). Refer to Implementing SSL and all
subsections applicable to your environment.
To create a TLS authentication method, use the command CREATE AUTHENTICATION as
documented in the SQL Reference Manual.
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
Note: The source code and installation instructions for oidentd are available at the oidentd
website.
2. Make sure that the Ident server accepts IPv6 connections to prevent authentication failure. To
do so, you must enable this capability. In the script /etc/init.d/oidentid, change the line
from:
exec="/usr/sbin/oidentd"
to
exec="/usr/sbin/oidentd -a ::"
2. Set the Ident authentication parameters, specifying the system users who should be allowed to
connect to your database.
Administrator's Guide
Implementing Security
3. Associate the authentication method with the HP Vertica user. Use a GRANTstatement that
allows the system user user1 to log in using Ident authentication:
=> GRANTAUTHENTICATIONv_ident TOuser1;
Set the system_users parameter to * to allow any system user to connect to the database as
vuser1:
=>CREATEAUTHENTICATION v_ident METHOD 'ident' LOCAL;
=>ALTERAUTHENTICATIONv_ident SETsystem_users='*';
=>GRANTAUTHENTICATIONv_ident TOvuser1;
=>ENABLEAUTHENTICATIONv_ident;
Administrator's Guide
Implementing Security
Using a GRANTstatement, associate the v_ident authentication with Public to allow system_
user1 to log into the database as any database user:
=>CREATEAUTHENTICATIONv_ident METHOD'ident' LOCAL;
=> ALTERAUTHENTICATIONv_ident SETsystem_users='system_user1';
=>GRANTAUTHENTICATIONv_ident to Public;
=>ENABLEAUTHENTICATIONv_ident;
Administrator's Guide
Implementing Security
Implementing SSL
To protect privacy and verify data integrity, you can configure HP Vertica and database clients to
use Secure Socket Layer (SSL). SSL allows secure connection and communication between the
client and the server. The SSL protocol uses a trusted third party called a Certificate Authority (CA).
Both the owner of a certificate and the party that relies on the certificate trust the CA.
HP Vertica supports the following authentication methods under SSL v3/Transport Layer Security
(TLS) 1.0 protocol:
l
SSL server authentication Lets the client confirm the server's identity. The client verifies
that the server's certificate and public key are valid and were issued by a certificate authority
(CA) listed in the client's list of trusted CAs. This authentication helps prevent man-in-themiddle attacks. See "Prerequisites for SSL Server Authentication and SSL Encryption" in SSL
Prerequisites and Configuring SSL.
SSL client authentication (Optional) Lets the server confirm the client's identity. The server
verifies that the client's certificate and public key are valid and were issued by a certificate
authority (CA) listed in the server's list of trusted CAs. Client authentication is optional because
HP Vertica can authenticate the client at the application protocol level through user name and
password credentials. See "Optional Prerequisites for SSL Server and Client Mutual
Authentication" in SSL Prerequisites.
Encryption Encrypts data sent between the client and database server. This method
significantly reduces the likelihood that the data can be read if the connection between the client
and server is compromised. Encryption works at both ends of a transaction, regardless of
whether SSL Client Authentication is enabled. See "Prerequisites for SSL Server Authentication
and SSL encryption" in SSL Prerequisites and Configuring SSL.
Data integrity Verifies that data sent between the client and server has not been altered
during transmission.
Note: For server authentication, HP Vertica supports using RSA encryption with ephemeral
Diffie-Hellman (DH). DH is the key agreement protocol.
Certificate Authority
The CA issues electronic certificates to identify one or both ends of a transaction. These
certificates to verify ownership of a public key by the name on the certificate.
Administrator's Guide
Implementing Security
If encrypted with a public key, data can be decrypted by its corresponding private key only.
If encrypted with a private key, data can be decrypted by its corresponding public key only.
For example, suppose Alice wants to send confidential data to Bob. Because she wants only Bob
to read it, she encrypts the data with Bob's public key. Even if someone else gains access to the
encrypted data, it remains protected. Because only Bob has access to his corresponding private
key, he is the only person who can decrypt Alice's encrypted data back into its original form.
SSL Prerequisites
Before you implement SSL security, obtain the appropriate certificate signed by a certificate
authority (CA) and private key files. Then, copy the certificate to your system. (See the OpenSSL
documentation.) These files must use the Privacy-Enhanced Mail (PEM) format.
If you are planning to use SSL mutual mode with JDBC, copy the root.crt file to one of the
clients; the root.crt is later incorporated into the truststore. The truststore can then be copied
to other clients. For more information, see JDBC Certificates in the section Generating SSL
Certificates and Keys.
Administrator's Guide
Implementing Security
If the VSQL_HOME environment variable is not set, copy the root.crt file to the .vsql
subdir of the login user's home directory. (e.g., ~/.vsql/root.crt)
If the VSQL_HOME environment variable is set, copy the root.crt file to the .vsql
subdir of the target directory. (e.g., $vsql_home/.vsql/root.crt)
The root.crt file contains either the server's certificate or the Certificate Authority that
issued the server certificate.
Important:If you do not perform this step, database operation may be compromised.
If the client cannot authenticate the server, the database does not start. This
situation can occur if a counterfeit server with the valid certificate file tries to
connect, the client detects that root.crt does not match the CA used to sign the
certificate.
If the VSQL_HOME environment variable is set, copy the file to the .vsql subdir of the target
directory. (e.g., $vsql_home/.vsql/client.crt)
If the VSQL_HOME environment variable is not set, copy the files to the .vsql subdir of the
login user's home directory. (e.g., ~/.vsql/client.crt)
Administrator's Guide
Implementing Security
If you are using either ODBC or JDBC, you can place the files anywhere on your system.
Then, provide the location in the connection string (ODBC/JDBC) or ODBCINI (ODBC only).
See Configuring SSL for ODBC Clients and Configuring SSL for JDBC Clients.
Important: If you're using ODBC, the private key file (client.key) must have read and
write permissions only for the dbadmin user (such as 0600 for Linux). Do not provide any
additional permissions or extend them to any other users.
Generating Certificate Authority (CA) certificates and keys that can then be used to sign server
and client keys.
Creating a server private key and requesting a new server certificate that includes a public key.
Creating a client private key and requesting a new client certificate that includes a public key.
For more detailed information on creating signed certificates, refer to the OpenSSL documentation.
Sample procedures show you how to create certifications and keys. All examples are hypothetical;
the commands shown allow many other possible options not used in these examples. Create your
commands based upon your specific environment.
Administrator's Guide
Implementing Security
You enter the following sample CA certificate values in response to openssl command line
prompts. Rather than enter these values from command line prompts, you can provide the
same information within .conf files (as shown in the command examples in this section).
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:Massachusetts
Locality Name (e.g., city) [Newbury]:Cambridge
Organization Name (e.g., company) [My Company Ltd]:HP Vertica
Organizational Unit Name (e.g., section) []:Support_CA
Common Name (e.g., your name or your server's hostname) []:myhost
Email Address []:[email protected]
When you create a certificate, you must include one unique name for each certificate that you
create. This unique name is referred to as a Distinguished Name (DN). The examples in this
section use the Organizational Unit Name for the DN.
Result: You now have a CA private key, new_servercakey.pem. You also have a CA public
certificate, new_serverca.crt. Use both the private key and the public certificate in the procedures
that follow for creating server and client certificates.
Administrator's Guide
Implementing Security
response to command-line prompts.) In this example, the Organizational Unit Name contains
the unique DN, Support_server.
openssl x509 -req -in new_server_reqout.txt -days 3650 -sha1 CAcreateserial -CA new_serverca.crt -CAkey new_servercakey.pem -out new_
server.crt
Result: You created the server private key file, new_server.key, and then signed the server
certificate using the CA private key (new_servercakey.pem) and CA public certificate (new_
serverca.crt). The result outputs to a new server certificate, new_server.crt.
Administrator's Guide
Implementing Security
response to command-line prompts. In this example, the Organizational Unit Name contains
the unique DN, Support_client.
Important: If you plan to use mutual authentication, you can create the client certificate in
a way that allows self-authentication. To do so, set the Common Name (CN) field to the
value of the database user name you want to use with the target database.
openssl x509 -req -in new_client_reqout.txt -days 3650 -sha1 CAcreateserial -CA new_serverca.crt -CAkey new_servercakey.pem -out new_
client.crt
Result: You created the client private key file, new_client.key, and then signed the client
certificate using the CA private key (new_servercakey.pem) and CA public certificate (new_
serverca.crt). The result outputs to a new server certificate, new_client.crt.
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
You do not need to set permissions for server certificates and keys because you add the contents
of those files to parameters on the server.
JDBC Certificates
The Java Runtime Environment (JRE) manages all keys and certificates with a keystore and a
truststore.
l
A keystore is a repository that includes private keys and trusted certificates. The private keys
have public key certificates associated with them. The keystore provides credentials for
authentication.
A truststore contains the certificates of trusted third-party certificate authorities with which you
communicate. The truststore verifies credentials.
After you generate the SSL certificates and keys, verify that the JRE detects the certificate
authoritys certificate.
This sample procedure adds a client certificate to a keystore/truststore.
1. Use the openssl x509 command to convert the CA certificate to a form that Java can detect.
The -noprompt option allows you to proceed without prompts. You can add the commands
given in this procedure to a script.
The alias and storepass values shown are arbitrary examples only. Specify values
appropriate for use in your environment.
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
Signed Certificates
For production, you must use certificates signed by a certificate authority. You can create and
submit a certificate.Then, when the certificate returns from the CA, import the certificate into MC.
To generate a new CSR, use the openssl command:
openssl req -new -key /opt/vertica/config/keystore.key -out server.csr
When you press Enter, you are prompted to enter information that will be incorporated into your
certificate request. Some fields contain a default value, which, for security reasons, you should
change. Other fields you can leave blank, such as password and optional company name. To leave
the field blank, type '.' .
Important: The keystore.key value for the -key option creates private key for the keystore. If
you generate a new key and import it using the Management Console interface, the MC
Administrator's Guide
Implementing Security
process does restart properly. You must restore the original keystore.jks file and restart
Management Console.
This information is contained in the CSR and shows both the default and replacement values:
Country Name (2 letter code) [GB]:USState or Province Name (full name)
[Berkshire]:Massachusetts
Locality Name (eg, city) [Newbury]: Cambridge
Organization Name (eg, company) [My Company Ltd]:HP
Organizational Unit Name (eg, section) []:Information Management
Common Name (eg, your name or your server's hostname) []:console.vertica.com
Email Address []:[email protected]
The Common Name field is the fully qualified domain name of your server. Your entry must
exactly match what you type in your web browser, or you receive a name mismatch error.
Self-Signed Certificates
To test your new SSL implementation, you can self-sign a CSR using either a temporary certificate
or your own internal CA, if one is available.
Note: A self-signed certificate generates a browser-based error notifying you that the signing
certificate authority is unknown and not trusted. For testing purposes, accept the risks and
continue.
The following command generates a temporary certificate, which expires after 365 days:
openssl x509 -req -days 365 -in server.csr -signkey /opt/vertica/config/keystore.key -out
server.crt
You can now import the self-signed key, server.crt, into Management Console.
See Also
l
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
Note: If you are upgrading to 7.1, the SSLCertificate and SSLPrivateKey parameters
are automatically set by Admintools if you set EnableSSL=1 in the previous version.
Important: If your server.crt SSL certificate file includes a certificate chain (more
than one certificate), Admintools automatically chooses the top-most certificate.
7. Configure SSL.
Configuring SSL
Configure SSL for each server in the cluster.
1. Verify that you have performed at least the minimum steps required in SSL Prerequisites for
server authentication and encryption and, optionally, for mutual authentication.
2. Next, verify that you have performed the steps listed in Distributing Certificates and Keys.
Important: Before you set the parameters SSLCertificate and SSLPrivateKey, you must
first set the EnableSSL parameter. Admintools sets these parameters for you when you
perform the procedure steps listed in Distributing Certificates and Keys. Alternatively, you
can use vsql to set the parameters using the ALTERDATABASEstatement. For more
information on setting configuration parameters see ALTER DATABASE.
These parameters are also automatically set during upgrade to 7.1 if you set EnableSSL=1
in the previous version.
3. Set the EnableSSL parameter to 1. By default, EnableSSL is set to 0 (disabled).
Administrator's Guide
Implementing Security
vsql automatically tries to connect using SSL. If a connection fails, and your server is started
in SSL Server Mode, vsql attempts to make a second connection over clear text. If you start
the server in SSL Mutual Mode, the connection fails without vsql attempting to connect over
clear text.
Linux and UNIX Enter the connection properties in the odbc.ini file. See Creating an ODBC
DSN for Linux, Solaris, AIX, and HP-UX Clients.
Microsoft Windows Enter the connection properties in the Windows Registry. See Creating
an ODBC DSN for Windows Clients.
For Windows ODBC, as of HP Vertica 7.1, you can set connection string properties within the
ODBC Data Source Administrator GUI, under the tab, Client Settings. Properties you can set
Administrator's Guide
Implementing Security
Description
require
Requires the server to use SSL. If the server cannot provide an encrypted
channel, the connection fails.
prefer
(Default value) Indicates your preference that the server to use SSL.The first
connection to the database tries to use SSL. If that connection fails, a second
connection is attempted over a clear channel.
allow
Makes a connection to the server whether the server uses SSL or not. The first
connection attempt to the database is made over a clear channel. If that
connection fails, a second connection is attempted over SSL.
disable
Never connects to the server using SSL. This setting is typically used for
troubleshooting.
Administrator's Guide
Implementing Security
Administrator's Guide
Implementing Security
You can use several debug specifiers (options) with the debug utility. The specifiers help narrow the
scope of the debugging information that is returned. For example, you could specify one of the
options that prints handshake messages or session activity.
For information on the debug utility and its options, see Debugging Utilities in the Oracle document,
JSSE Reference Guide.
For information on interpreting debug information, refer to the Oracle document, Debugging
SSL/TLS Connections.
Administrator's Guide
Managing Users and Privileges
See Also
l
GRANT Statements
REVOKE Statements
Administrator's Guide
Managing Users and Privileges
See Also
l
CREATE USER
About MC Users
Administrator's Guide
Managing Users and Privileges
Object owner
Note: External to an HP Vertica database, an MC administrator can create users through the
Management Console and grant them database access. See About MC Users for details.
DBADMIN User
When you create a new database, a single database administrator account, DBADMIN, is
automatically created along with the PUBLIC role. This database superuser bypasses all
permission checks and has the authority to perform all database operations, such as bypassing all
GRANT/REVOKE authorizations and any user granted the PSEUDOSUPERUSER role.
Note: Although the dbadmin user has the same name as the Linux database administrator
account, do not confuse the concept of a database superuser with Linux superuser (root)
privilege; they are not the same. A database superuser cannot have Linux superuser privileges.
The DBADMIN user can start and stop a database without a database password. To connect to the
database, a password is required.
See Also
l
DBADMIN Role
PSEUDOSUPERUSER Role
PUBLIC Role
Object Owner
An object owner is the user who creates a particular database object and can perform any operation
on that object. By default, only an owner (or a superuser) can act on a database object. In order to
Administrator's Guide
Managing Users and Privileges
allow other users to use an object, the owner or superuser must grant privileges to those users
using one of the GRANT statements.
Note: Object owners are PUBLIC users for objects that other users own.
See About Database Privileges for more information.
PUBLIC User
All non-DBA (superuser) or object owners are PUBLIC users.
Note: Object owners are PUBLIC users for objects that other users own.
Newly-created users do not have access to schema PUBLIC by default. Make sure to GRANT
USAGE ON SCHEMA PUBLIC to all users you create.
See Also
l
PUBLIC Role
Notes
l
Newly-created users do not have access to schema PUBLIC by default. Make sure to GRANT
USAGE ON SCHEMA PUBLIC to all users you create
By default, database users have the right to create temporary tables in the database.
If you plan to create users on Management Console, the database user account needs to exist
before you can associate an MC user with the database.
You can change information about a user, such as his or her password, by using the ALTER
USER statement. If you want to configure a user to not have any password authentication, you
Administrator's Guide
Managing Users and Privileges
can set the empty password in CREATE or ALTER USER statements, or omit the
IDENTIFIED BY parameter in CREATE USER.
Example
The following series of commands add user Fred to a database with password 'password. The
second command grants USAGE privileges to Fred on the public schema:
=> CREATE USER Fred IDENTIFIED BY 'password';=> GRANT USAGE ON SCHEMA PUBLIC to Fred;
User names created with double-quotes are case sensitive. For example:
=> CREATE USER "FrEd1";
In the above example, the logon name must be an exact match. If the user name was created
without double-quotes (for example, FRED1), then the user can log on as FRED1, FrEd1, fred1,
and so on.
ALTER USER and DROP USER syntax is not case sensitive.
See Also
l
Creating an MC User
To grant Fred database access, use UNLOCK syntax with the ALTER USER command:
=> ALTER USER Fred ACCOUNT UNLOCK;
Administrator's Guide
Managing Users and Privileges
=> \c - Fred
You are now connected as user "Fred".
CREATE USER also supports UNLOCK syntax; however, UNLOCK is the default, so you don't
need to specify the keyword when you create a new user to whom you want to grant immediate
database access.
Administrator's Guide
Managing Users and Privileges
Non-LDAP passwords on MC are for MC access only and are not related to a user's logon
credentials on the HP Vertica database.
1. Sign in to Management Console and navigate to MC Settings > User management.
2. Click to select the user to modify and click Edit.
3. Click Edit password and enter the new password twice.
4. Click OK and then click Save.
Administrator's Guide
Managing Users and Privileges
About MC Users
Unlike database users, which you create on the HP Vertica database and then grant privileges and
roles through SQL statements, you create MC users on the Management Console interface. MC
users are external to the database; their information is stored on an internal database on the MC
application/web server, and their access to both MC and to MC-managed databases is controlled
by groups of privileges (also referred to as access levels). MC users are not system (Linux) users;
they are entries in the MC internal database.
MC configuration privileges are made up of roles that control what users can configure on the
MC, such as modify MC settings, create/import HP Vertica databases, restart MC, create an
HP Vertica cluster through the MC interfac, and create and manage MC users.
MC database privileges are made up of roles that control what users can see or do on an MCmanaged HP Vertica database, such as view the database cluster state, query and session
activity, monitor database messages and read log files, replace cluster nodes, and stop
databases.
If you are using MC, you might want to allow one or more users in your organization to configure and
manage MC, and you might want other users to have database access only. You can meet these
requirements by creating MC users and granting them a role from each privileges group. See
Creating an MC User for details.
MC User Types
There are four types of role-based users on MC:
l
The default superuser administrator (Linux account) who gets created when you install and
configure MC and oversees the entire MC. See SUPER Role (mc).
Users who can configure all aspects of MC and control all MC-managed databases. See
ADMIN Role (mc).
Users who can configure some aspects of MC and monitor all MC-managed databases. See IT
Role (mc).
Administrator's Guide
Managing Users and Privileges
Users who cannot configure MC and have access to one or more MC-managed databases only.
See NONE Role (mc).
You create users and grant them privileges (through roles) on the MC Settings page by selecting
User management, to add users who will be authenticated against the MC or Authentication, to
authenticate MC users through your organization's LDAP repository.
To add users who will be authenticated against the MC, click User Management
To add users who will be authenticated through your organization's LDAP repository, click
Authentication
MC supports only one method for authentication, so if you choose MC, all MC users will be
authenticated using their MC login credentials.
Default MC Users
The MC super account is the only default user. The super or another MC administrator must create
all other MC users.
See Also
l
Management Console
Creating an MC User
MC provides two authentication schemes for MC users: LDAP or MC (internal). Which method you
choose will be the method MC uses to authenticate all MC users. It is not possible to authenticate
some MC users against LDAP and other MC users against credentials in the database through MC.
Administrator's Guide
Managing Users and Privileges
MC (internal) authentication. Internal user authorization is specific to the MC itself, where you
create a user with a username and password combination. This method stores MC user
information in an internal database on the MC application/web server, and encrypts passwords.
Note that these MC users are not system (Linux) users; they are entries in the MCs internal
database.
LDAP authentication. All MC usersexcept for the MC super administrator, which is a Linux
accountwill be authenticated based on search criteria against your organization's LDAP
repository. MC uses information from LDAP for authentication purposes only and does not
modify LDAP information. Also, MC does not store LDAP passwords but passes them to the
LDAP server for authentication.
If you chose MC authentication, follow the instructions under Create a new MC-authenticated
user.
If you chose LDAP authentication, follow the instructions under Create a new user from
LDAP.
Prerequisites
Before you create an MC user, you already:
l
Created a database directly on the server or through the MC interface, or you imported an
existing database cluster into the MC interface. See Managing Database Clusters on MC.
Created a database user account (source user) on the server, which has the privileges and/or
roles you want to map to the new (target) MC user. See Creating a Database User.
Know what MC privileges you want to grant the new MC user. See About MC Privileges and
Roles.
If you have not yet met the first two above prerequisites, you can still create new MC users; you
just won't be able to map them to a database until after the database and target database user exist.
To grant MC users database access later, see Granting Database Access to MC Users.
Administrator's Guide
Managing Users and Privileges
Note: It is not necessary to give the MC user the exact same name as the database user
account you'll map the MC user to in Step 7. What matters is that the source database
user has privileges and/or roles similar to the database role you want to grant the MC user.
The most likely scenario is that you will map multiple MC users to a single database user
account. See MC Database Privileges and Mapping an MC User to a Database User's
Privileges for more information.
4. Let MC generate a password or create one by clicking Edit password. If LDAP has been
configured, the MC password field will not appear.
5. Optionally enter the user's e-mail address.
6. Select an MC configuration permissions level. See MC Configuration Privileges.
7. Next to the DB access levels section, click Add to grant this user database permissions. If
you want to grant access later, proceed to Step 8. If you want to grant database access now,
provide the following information:
i. Choose a database. Select a database from the list of MC-discovered (databases
that were created on or imported into the MC interface).
ii. Database username. Enter an existing database user name or, if the database is
running, click the ellipses [] to browse for a list of database users, and select a
name from the list.
iii. Database password. Enter the password to the database user account (not this
username's password).
iv. Restricted access. Chose a database level (ADMIN, IT, or USER) for this user.
v. Click OK to close the Add permissions dialog box.
Administrator's Guide
Managing Users and Privileges
See Mapping an MC User to a Database User's Privileges for additional information about
associating the two user accounts.
8. Leave the user's Status as enabled (the default). If you need to prevent this user from
accessing MC, select disabled.
9. Click Add User to finish.
Administrator's Guide
Managing Users and Privileges
organization's LDAP directory. If the credentials are found to be invalid, you are asked to re-enter
them.
If the database is not available at the time you create the new user, MC saves the
username/password and prompts for validation when the user accesses the Database and Clusters
page later.
See Also
l
Configuring MC
About MC Users
Managing MC Users
You can manage MC users using the User Management tab on the MCSettings page.
Administrator's Guide
Managing Users and Privileges
MC password
Email address. This field is optional, When the user is authenticated against LDAP, the email
field is auto-populated with that user's email address if one exists.
You can change a user's status by enabling or disabling access to MC. You can also delete users.
See Also
l
About MC Users
Administrator's Guide
Managing Users and Privileges
Create a schema
Create a view
Run procedures
Before HP Vertica executes a statement, it determines if the requesting user has the necessary
privileges to perform the operation.
For more information about the privileges associated with these resources, see Privileges That Can
Be Granted on Objects.
Note: HP Vertica logs information about each grant (grantor, grantee, privilege, and so on) in
the V_CATALOG.GRANTS system table.
See Also
l
GRANT Statements
REVOKE Statements
Administrator's Guide
Managing Users and Privileges
Schemas
The PUBLIC schema is present in any newly-created HP Vertica database, and newly-created
users have only USAGE privilege on PUBLIC. A database superuser must explicitly grant new
users CREATE privileges, as well as grant them individual object privileges so the new users can
create or look up objects in the PUBLIC schema.
Operation
Required Privileges
CREATE SCHEMA
DROP SCHEMA
Schema owner
Administrator's Guide
Managing Users and Privileges
Tables
Operation
Required Privileges
CREATE TABLE
DROP TABLE
TRUNCATE TABLE
SELECT
INSERT
Administrator's Guide
Managing Users and Privileges
Operation
DELETE
Required Privileges
l
UPDATE
REFERENCES
ANALYZE_STATISTICS()
ANALYZE_HISTOGRAM()
DROP_STATISTICS()
DROP_PARTITION()
MERGE_PARTITIONS()
Administrator's Guide
Managing Users and Privileges
Views
Operation
CREATE VIEW
DROP VIEW
SELECT ... FROM VIEW
Required Privileges
l
Note: Privileges required on base objects for view owner must be directly
granted, not through roles:
l
View owner must have SELECT ... WITH GRANT OPTION privileges
on the view's base tables or views if non-owner runs a SELECT query
on the view. This privilege must be directly granted to the owner,not
through a role.
View owner must have SELECT privilege directly granted (not through
a role) on a view's base objects (table or view) if owner runs a
SELECT query on the view.
Projections
Operation
CREATE PROJECTION
Required Privileges
l
Administrator's Guide
Managing Users and Privileges
Operation
Required Privileges
AUTO/DELAYED PROJECTION
DROP PROJECTION
External Procedures
Operation
Required Privileges
CREATE PROCEDURE
Superuser
DROP PROCEDURE
Superuser
EXECUTE
Libraries
Operation
Required Privileges
CREATE LIBRARY
Superuser
DROP LIBRARY
Superuser
User-Defined Functions
The following abbreviations are used in the UDF table:
l
UDF = Scalar
UDT = Transform
Administrator's Guide
Managing Users and Privileges
UDAnF= Analytic
UDAF = Aggregate
Operation
CREATE
CREATE
CREATE
CREATE
Required Privileges
Sequences
Operation
Required Privileges
CREATE SEQUENCE
Administrator's Guide
Managing Users and Privileges
Operation
Required Privileges
DROP SEQUENCE
CURRVAL()NEXTVAL()
Resource Pools
Operation
Required Privileges
Superuser
MAXMEMORYSIZE
PRIORITY
QUEUETIMEOUT
PLANNEDCONCURRENCY
SINGLEINITIATOR
MAXCONCURRENCY
Users can only change their own resource pool setting using
ALTER USER syntax
Superuser
Administrator's Guide
Managing Users and Privileges
Users/Profiles/Roles
Operation
Required Privileges
CREATE USER
Superuser
CREATE PROFILE
CREATE ROLE
ALTER USER
Superuser
ALTER PROFILE
ALTER ROLE RENAME
DROP USER
Superuser
DROP PROFILE
DROP ROLE
Object Visibility
You can use one or a combination of vsql \d [pattern] meta commands and SQL system tables to
view objects on which you have privileges to view.
l
Use \dt [pattern] to view all tables in the database, as well as the system table V_
CATALOG.TABLES
Use \dj [pattern] to view projections showing the schema, projection name, owner, and node, as
well as the system table V_CATALOG.PROJECTIONS
Operation
Required
Privileges
Look up schema
At least one
privilege on
schema that
contains the
object
Administrator's Guide
Managing Users and Privileges
Operation
Required
Privileges
USAGE
privilege on
schema
At least one
privilege on
any of the
following
objects:
TABLE
VIEW
FUNCTION
PROCEDURE
SEQUENCE
Look up Projection
At least one
privilege on all
base tables
USAGE
privilege on
schema of all
base table
SELECT
privilege on the
resource pool
Existence of object
USAGE
privilege on the
schema that
contains the
object
Administrator's Guide
Managing Users and Privileges
I/O Operations
Operation
Required Privileges
CONNECTDISCONNECT
None
EXPORT TO HP Vertica
COPY LOCAL
Superuser
l
Administrator's Guide
Managing Users and Privileges
Comments
Operation
Required Privileges
AGGREGATE
FUNCTION
ANALYTIC FUNCTION
COLUMN
CONSTRAINT
FUNCTION
LIBRARY
NODE
PROJECTION
SCHEMA
SEQUENCE
TABLE
TRANSFORM
FUNCTION
VIEW
Transactions
Operation
Required Privileges
COMMIT
None
ROLLBACK
None
RELEASE SAVEPOINT
None
SAVEPOINT
None
Administrator's Guide
Managing Users and Privileges
Sessions
Operation
Required Privileges
None
DATESTYLE
ESCAPE_STRING_WARNING
INTERVALSTYLE
LOCALE
ROLE
SEARCH_PATH
SESSION AUTOCOMMIT
SESSION CHARACTERISTICS
SESSION MEMORYCAP
SESSION RUNTIMECAP
SESSION TEMPSPACE
STANDARD_CONFORMING_
STRINGS
TIMEZONE
None
Tuning Operations
Operation Required Privileges
PROFILE
EXPLAIN
Same privileges required to run the query for which you use the EXPLAIN keyword
Administrator's Guide
Managing Users and Privileges
See Also
l
GRANT Statements
REVOKE Statements
Database Privileges
Only a database superuser can create a database. In a new database, the PUBLIC Role is granted
USAGE on the automatically-created PUBLIC schema. It is up to the superuser to grant further
privileges to users and roles.
The only privilege a superuser can grant on the database itself is CREATE, which allows the user
to create a new schema in the database. For details on granting and revoking privileges on a
database, see the GRANT (Database) and REVOKE (Database) topics in the SQL Reference
Manual.
Privilege Grantor
CREATE
Description
Administrator's Guide
Managing Users and Privileges
Schema Privileges
By default, only a superuser and the schema owner have privileges to create objects within a
schema. Additionally, only the schema owner or a superuser can drop or alter a schema. See
DROP SCHEMA and ALTER SCHEMA.
You must grant all new users access to the PUBLIC schema by running GRANT USAGE ON
SCHEMA PUBLIC. Then grant new users CREATEprivileges and privileges to individual objects
in the schema. This enables new users to create or locate objects in the PUBLICschema. Without
USAGE privilege, objects in the schema cannot be used or altered, even by the object owner.
CREATE gives the schema owner or user WITH GRANT OPTION permission to create new
objects in the schema, including renaming an object in the schema or moving an object into this
schema.
Note: The schema owner is typically the user who creates the schema. However, a superuser
can create a schema and assign ownership of the schema to a different user at creation.
All other access to the schema and its objects must be explicitly granted to users or roles by the
superuser or schema owner. This prevents unauthorized users from accessing the schema and its
objects. A user can be granted one of the following privileges through the GRANT statement:
Privilege Description
CREATE
Allows the user to create new objects within the schema. This includes the ability to
create a new object, rename existing objects, and move objects into the schema
from other schemas.
USAGE
Permission to select, access, alter, and drop objects in the schema. The user must
also be granted access to the individual objects in order to alter them. For example, a
user would need to be granted USAGE on the schema and SELECT on a table to be
able to select data from a table. You receive an error message if you attempt to query
a table that you have SELECT privileges on, but do not have USAGE privileges for
the schema that contains the table.
Note the following on error messages related to granting privileges on a schema or an object:
l
You attempt to grant a privilege to a schema, but you do not have USAGE privilege for the
schema. In this case, you receive an error message that the schema does not exist.
You attempt to grant a privilege to an object within a schema, and you have USAGE privilege on
Administrator's Guide
Managing Users and Privileges
the schema. You do not have privilege on the individual object within the schema. In this case,
you receive an error denying permission for that object.
Creating an object
Accessing/altering an
object
create the object in the current schema (the first schema in the
schema search path), returning an error if the schema does not exist
statements search
Note: The first schema in the search path is the current schema, and
statement.
name.
l
Administrator's Guide
Managing Users and Privileges
See Also
l
GRANT (Schema)
REVOKE (Schema)
Table Privileges
By default, only a superuser and the table owner (typically the person who creates a table) have
access to a table. The ability to drop or alter a table is also reserved for a superuser or table owner.
This privilege cannot be granted to other users.
All other users or roles (including the user who owns the schema, if he or she does not also own the
table) must be explicitly granted using WITH GRANT OPTION syntax to access the table.
These are the table privileges a superuser or table owner can grant:
Privilege
Description
SELECT
INSERT
DELETE
Permission to DELETE data from the table, as well as SELECT privilege on the
table when executing a DELETE statement that references table column values in a
WHERE or SET clause.
UPDATE
Permission to UPDATE and change data in the table, as well as SELECT privilege
on the table when executing an UPDATE statement that references table column
values in a WHERE or SET clause.
REFERENCES
To use any of the above privileges, the user must also have USAGE privileges on the schema that
contains the table. See Schema Privileges for details.
Referencing sequence in the CREATE TABLE statement requires the following privileges:
l
Administrator's Guide
Managing Users and Privileges
For details on granting and revoking table privileges, see GRANT (Table) and REVOKE (Table) in
the SQL Reference Manual.
Projection Privileges
Because projections are the underlying storage construct for tables, they are atypical in that they
do not have an owner or privileges associated with them directly. Instead, the privileges to create,
access, or alter a projection are based on the anchor and base tables that the projection references,
as well as the schemas that contain them.
To be able run a query involving a projection, a user must have SELECT privileges on the table or
tables that the projection references, and USAGE privileges on all the schemas that contain those
tables.
There are two ways to create projection: explicitly and implicitly.
USAGE privilege on all the schemas that contain the base tables referenced by the projection
Only the anchor table owner can drop explicitly created projections or pre-join projections.
Explicitly created projections can be live aggregate projections, including Top-K projections and
projections with expressions.
To prevent data loss and inconsistencies, tables must contain one superprojection. You cannot
drop a projection if that projection is the table's only superprojection.
You cannot drop a buddy projection if dropping that projection violates the system K-safety .
Administrator's Guide
Managing Users and Privileges
HP Vertica does not associate privileges directly with projections since they are the underlying
storage construct. Privileges may only be granted on the logical storage containers: the tables and
views.
Dropping Projections
Dropping projections are handled much the same way HP Vertica creates them:
l
View Privileges
A View is a stored query that dynamically accesses and computes data from the database at
execution time. Use /dv in vsql to display available views. By default, the only the following have
privileges to access the base object for a view:
l
a superuser
the View owner (typically the person who creates the view)
USAGE privileges on the schema in which the View resides assigned with GRANT(Schema).
You can assign View privileges to other users and roles using GRANT (View), for example:
l
Administrator's Guide
Managing Users and Privileges
See Also
GRANT (View)
REVOKE (View)
Sequence Privileges
To create a sequence, a user must have CREATE privileges on schema that contains the
sequence. Only the owner and superusers can initially access the sequence. All other users must
be granted access to the sequence by a superuser or the owner.
Only the sequence owner (typically the person who creates the sequence) or can drop or rename a
sequence, or change the schema in which the sequence resides:
l
DROP SEQUENCE: Only a sequence owner or schema owner can drop a sequence.
ALTER SEQUENCE RENAME TO: A sequence owner must have USAGE and CREATE
privileges on the schema that contains the sequence to be renamed.
ALTER SEQUENCE SET SCHEMA: A sequence owner must have USAGE privilege on the
schema that currently contains the sequence (old schema), as well as CREATE privilege on the
schema where the sequence will be moved (new schema).
The following table lists the privileges that can be granted to users or roles on sequences.
The only privilege that can be granted to a user or role is SELECT, which allows the user to use
CURRVAL() and NEXTVAL() on sequence and reference in table. The user or role also needs to
have USAGE privilege on the schema containing the sequence.
Privilege Description
SELECT
USAGE
Note: Referencing sequence in the CREATE TABLE statement requires SELECT privilege on
sequence object and USAGE privilege on sequence schema.
For details on granting and revoking sequence privileges, see GRANT (Sequence) and REVOKE
(Sequence) in the SQL Reference Manual.
Administrator's Guide
Managing Users and Privileges
See Also
l
USAGE
For details on granting and revoking external table privileges, see GRANT (Procedure) and
REVOKE (Procedure) in the SQL Reference Manual.
USAGE
DROP FUNCTION: Only a superuser or function owner can drop the function.
ALTER FUNCTION RENAME TO: A superuser or function owner must have USAGE and
CREATE privileges on the schema that contains the function to be renamed.
Administrator's Guide
Managing Users and Privileges
ALTER FUNCTION SET SCHEMA: A superuser or function owner must have USAGE privilege
on the schema that currently contains the function (old schema), as well as CREATE privilege
on the schema where the function will be moved (new schema).
For details on granting and revoking user-defined function privileges, see the following topics in the
SQL Reference Manual:
l
Library Privileges
Only a superuser can load an external library using the CREATE LIBRARY statement. By default,
only a superuser can create user-defined functions (UDFs) based on a loaded library. A superuser
can use the GRANT USAGE ON LIBRARY statement to allow users to create UDFs based on
classes in the library. The user must also have CREATE privileges on the schema that will contain
the UDF.
Privilege Description
USAGE
Once created, only a superuser or the user who created a UDF can use it by default. Either of them
can grant other users or roles the ability to call the function using the GRANT EXECUTE ON
FUNCTION statement. See the GRANT (User Defined Extension) and REVOKE (User Defined
Extension) topics in the SQL Reference Manual for more information on granting and revoking
privileges on functions.
In addition to EXECUTE privilege, users/roles also require USAGE privilege on the schema in
which the function resides in order to execute the function.
For more information about libraries and UDFs, see Developing and Using User Defined Functions
in the Extending HPVertica Guide.
Administrator's Guide
Managing Users and Privileges
RESOURCE POOL statement and the RESOURCE POOL clause of the ALTER USER statement
to have their queries draw their resources from the new pool.
Privilege Description
USAGE
SELECT
UPDATE
For details on granting and revoking resource pool privileges, see GRANT (Resource Pool) and
REVOKE (Resource Pool) in the SQL Reference Manual.
Once such storage locations exist and the appropriate privileges are granted, users and roles
granted READ privileges can copy data from files in the storage location into a table. Those granted
WRITE privileges can export data from a table to the storage location on which they have been
granted access. WRITE privileges also let users save COPY statement exceptions and rejected
data files from HP Vertica to the specified storage location.
Only a superuser can add, alter, retire, drop, and restore a location, as well as set and measure
location performance. All non-dbadmin users or roles require READ and/or WRITE permissions on
the location.
Privilege Description
READ
Allows the user to copy data from files in the storage location into a table.
Administrator's Guide
Managing Users and Privileges
Privilege Description
WRITE
Allows the user to copy data to the specific storage location. Users with WRITE
privileges can also save COPY statement exceptions and rejected data files to the
specified storage location.
See Also
l
CREATE LOCATION
role
profile
user
By default, only the superuser can grant or revoke a role to another user or role. A user or role can be
given the privilege to grant and revoke a role by using the WITH ADMIN OPTION clause of the
GRANT statement.
For details on granting and revoking role privileges, see GRANT (Role) and REVOKE (Role) in the
SQL Reference Manual.
See Also
l
CREATE USER
ALTER USER
DROP USER
CREATE PROFILE
ALTER PROFILE
Administrator's Guide
Managing Users and Privileges
DROP PROFILE
CREATE ROLE
DROP ROLE
Metadata Privileges
A superuser has unrestricted access to all database metadata. Other users have significantly
reduced access to metadata based on their privileges, as follows:
Type of
Metadata
User Access
Catalog
Users must possess USAGE privilege on the schema and any type of access
objects:
(SELECT) or modify privilege on the object to see catalog metadata about the
object. See also Schema Privileges.
Tables
For internal objects like projections, WOS and ROS containers that don't have
Columns
Constraints
Sequences
access privileges directly associated with them, the user must possess the
requisite privileges on the associated schema and table objects instead. For
example, to see whether a table has any data in the WOS, you need to have
USAGE on the table schema and at least SELECT on the table itself. See also
Table Privileges and Projection Privileges.
External
Procedures
Projections
ROS
containers
WOS
Administrator's Guide
Managing Users and Privileges
Type of
Metadata
User Access
User sessions
Users can only access information about their own, current sessions.
and functions,
and system
tables related
CURRENT_DATABASE
CURRENT_SCHEMA
CURRENT_USER
HAS_TABLE_PRIVILEGE
to these
sessions
I/O Privileges
Users need no special permissions to connect to and disconnect from an HP Vertica database.
To EXPORT TO and COPY FROM HP Vertica, the user must have:
l
To COPY FROM STDIN and use local COPY a user must have INSERT privileges on the table
and USAGE privilege on schema.
Note: Only a superuser can COPY from file.
Administrator's Guide
Managing Users and Privileges
Comment Privileges
A comment lets you add, revise, or remove a textual message to a database object. You must be
an object owner or superuser in order to COMMENT ON one of the following objects:
l
COLUMN
CONSTRAINT
LIBRARY
NODE
PROJECTION
SCHEMA
SEQUENCE
TABLE
TRANSFORM FUNCTION
VIEW
Other users must have VIEW privileges on an object to view its comments.
Transaction Privileges
No special permissions are required for the following database operations:
l
COMMIT
ROLLBACK
RELEASE SAVEPOINT
SAVEPOINT
Administrator's Guide
Managing Users and Privileges
Session Privileges
No special permissions are required for users to use the SHOW statement or any of the SET
statements.
Tuning Privileges
In order to PROFILE a single SQL statement or returns a query plan's execution strategy to
standard output using the EXPLAIN command, users must have the same privileges that are
required for them to run the same query without the PROFILE or EXPLAIN keyword.
The syntax for granting and revoking privileges is different for each database object, such as
schema, database, table, view, sequence, procedure, function, resource pool, and so on.
Normally, a superuser first creates a user and then uses GRANT syntax to define the user's
privileges or roles or both. For example, the following series of statements creates user Carol and
grants Carol access to the apps database in the PUBLIC schema and also lets Carol grant
SELECT privileges to other users on the applog table:
=>
=>
=>
=>
See GRANT Statements and REVOKE Statements in the SQL Reference Manual.
Administrator's Guide
Managing Users and Privileges
Note: Database superusers are not the same as a Linux superuser with (root) privilege and
cannot have Linux superuser privilege.
A superuser can grant privileges on all database object types to other users, as well as grant
privileges to roles. Users who have been granted the role will then gain the privilege as soon as
they enable it.
Superusers may grant or revoke any object privilege on behalf of the object owner, which means a
superuser can grant or revoke the object privilege if the object owner could have granted or revoked
the same object privilege. A superuser may revoke the privilege that an object owner granted, as
well as the reverse.
Since a superuser is acting on behalf of the object owner, the GRANTOR column of V_
CATALOG.GRANTS table displays the object owner rather than the superuser who issued the
GRANT statement.
A superuser can also alter ownership of table and sequence objects.
See Also
DBADMIN Role
Administrator's Guide
Managing Users and Privileges
Object owners must have appropriate schema privilege to access, alter, rename, move or drop any
object it owns without any additional privileges.
An object owner can also:
l
A superuser or object owner can grant privileges to roles. Users who have been granted the role
then gain the privilege.
An object owner can grant privileges on the object to other users using the optional WITH
GRANT OPTION clause.
The user needs to have USAGE privilege on schema and appropriate privileges on the object.
When a user grants an explicit list of privileges, such as GRANT INSERT, DELETE, REFERENCES ON
applog TO Bob:
l
The GRANT statement succeeds only if all the roles are granted successfully. If any grant
operation fails, the entire statement rolls back.
HP Vertica will return ERROR if the user does not have grant options for the privileges listed.
When a user grants ALL privileges, such as GRANT ALL ON applog TO Bob, the statement always
succeeds. HP Vertica grants all the privileges on which the grantor has the WITH GRANT OPTION
and skips those privileges without the optional WITH GRANT OPTION.
For example, if the user Bob has delete privileges with the optional grant option on the applog table,
only DELETE privileges are granted to Bob, and the statement succeeds:
Administrator's Guide
Managing Users and Privileges
For details, see the GRANT Statements in the SQL Reference Manual.
The above REVOKE statement removes the privilege (and WITH GRANT OPTION privilege, if
applicable) from the grantee or it prints a notice that the operation failed.
In order to revoke grant option for a privilege, the grantor must have previously granted the grant
option for the privilege to the specified grantee. Otherwise, HP Vertica prints a NOTICE.
The following REVOKE statement removes the GRANT option only but leaves the privilege intact:
=> GRANT INSERT on applog TO Bob WITH GRANT OPTION;
GRANT PRIVILEGE
=> REVOKE GRANT OPTION FOR INSERT ON applog FROM Bob;
REVOKE PRIVILEGE
When a user revokes an explicit list of privileges, such as GRANT INSERT, DELETE, REFERENCES
ON applog TO Bob:
l
The REVOKE statement succeeds only if all the roles are revoked successfully. If any revoke
operation fails, the entire statement rolls back.
HP Vertica returns ERROR if the user does not have grant options for the privileges listed.
HP Vertica returns NOTICE when revoking privileges that this user had not been previously
granted.
Administrator's Guide
Managing Users and Privileges
When a user revokes ALL privileges, such as REVOKE ALL ON applog TO Bob, the statement
always succeeds. HP Vertica revokes all the privileges on which the grantor has the optional WITH
GRANT OPTION and skips those privileges without the WITH GRANT OPTION.
For example, if the user Bob has delete privileges with the optional grant option on the applog table,
only grant option is revoked from Bob, and the statement succeeds without NOTICE:
=> REVOKE GRANT OPTION FOR DELETE ON applog FROM Bob;
For details, see the REVOKE Statements in the SQL Reference Manual.
User1 creates new table t1 within schema s1 and then grants SELECT WITH GRANT OPTION
privilege on s1.t1 to User2:
Administrator's Guide
Managing Users and Privileges
Users cannot revoke privileges upstream in the chain. For example, User2 did not grant privileges
on User1, so when User1 runs the following REVOKE command, HP Vertica rolls back the
command:
=> \c - User2You are now connected as user "User2".
=> REVOKE CREATE ON SCHEMA s1 FROM User1;
ROLLBACK 0: "CREATE" privilege(s) for schema "s1" could not be revoked from "User1"
Users can revoke privileges indirectly from users who received privileges through a cascading
chain, like the one shown in the example above. Here, users can use the CASCADE option to
revoke privileges from all users "downstream" in the chain. A superuser or User1 can use the
CASCADE option to revoke the SELECT privilege on table s1.t1 from all users. For example, a
superuser or User1 can execute the following statement to revoke the SELECT privilege from all
users and roles within the chain:
=> \c - User1You are now connected as user "User1".
=> REVOKE SELECT ON s1.t1 FROM User2 CASCADE;
REVOKE PRIVILEGE
When a superuser or User1 executes the above statement, the SELECT privilege on table s1.t1 is
revoked from User2, User3, and the reviewer role. The GRANT privilege is also revoked from
User2 and User3, which a superuser can verify by querying the V_CATALOG.GRANTS system
table.
=> SELECT * FROM grants WHERE object_name = 's1' AND grantee ILIKE 'User%';
Administrator's Guide
Managing Users and Privileges
Administrator's Guide
Managing Users and Privileges
Modifying Privileges
A superuser or object owner can use one of the ALTER statements to modify a privilege, such as
changing a sequence owner or table owner. Reassignment to the new owner does not transfer
grants from the original owner to the new owner; grants made by the original owner are dropped.
In order to alter table ownership, you must be either the table owner or a superuser.
A change in table ownership transfers just the owner and not privileges; grants made by the original
owner are dropped and all existing privileges on the table are revoked from the previous owner.
However, altering the table owner transfers ownership of dependent sequence objects (associated
IDENTITY/AUTO-INCREMENT sequences) but does not transfer ownership of other referenced
sequences. See ALTER SEQUENCE for details on transferring sequence ownership.
Notes
l
Table privileges are separate from schema privileges; therefore, a table privilege change or table
owner change does not result in any schema privilege change.
Because projections define the physical representation of the table, HP Vertica does not require
separate projection owners. The ability to create or drop projections is based on the table
privileges on which the projection is anchored.
During the alter operation HP Vertica updates projections anchored on the table owned by the
old owner to reflect the new owner. For pre-join projection operations, HP Vertica checks for
privileges on the referenced table.
Example
In this example, user Bob connects to the database, looks up the tables, and transfers ownership of
table t33 from himself to to user Alice.
Administrator's Guide
Managing Users and Privileges
Notice that when Bob looks up database tables again, he no longer sees table t33.
=> \d
List of tables
List of tables
Schema | Name | Kind | Owner | Comment
--------+--------+-------+---------+--------public | applog | table | dbadmin |
(1 row)
When user Alice connects to the database and looks up tables, she sees she is the owner of table
t33.
=> \c - AliceYou are now connected as user "Alice".
=> \d
List of tables
Schema | Name | Kind | Owner | Comment
--------+------+-------+-------+--------public | t33 | table | Alice |
(2 rows)
Either Alice or a superuser can transfer table ownership back to Bob. In the following case a
superuser performs the transfer.
=> \c - dbadminYou are now connected as user "dbadmin".
=> ALTER TABLE t33 OWNER TO Bob;
ALTER TABLE
=> \d
List of tables
Schema |
Name
| Kind | Owner | Comment
--------+----------+-------+---------+--------public | applog
| table | dbadmin |
public | comments | table | dbadmin |
public | t33
| table | Bob
|
s1
| t1
| table | User1
|
(4 rows)
You can also query the V_CATALOG.TABLES system table to view table and owner information.
Note that a change in ownership does not change the table ID.
In the below series of commands, the superuser changes table ownership back to Alice and queries
the TABLES system table.
Administrator's Guide
Managing Users and Privileges
Now the superuser changes table ownership back to Bob and queries the TABLES table again.
Nothing changes but the owner_name row, from Alice to Bob.
=> ALTER TABLE t33 OWNER TO Bob;
ALTER TABLE
=> SELECT table_schema_id, table_schema, table_id, table_name, owner_id, owner_name FROM
tables;
table_schema_id | table_schema |
table_id
| table_name |
owner_id
|
owner_name-------------------+--------------+-------------------+------------+------------------+-----------45035996273704968 | public
| 45035996273713634 | applog
| 45035996273704962 |
dbadmin
45035996273704968 | public
| 45035996273724496 | comments
| 45035996273704962 |
dbadmin
45035996273730528 | s1
| 45035996273730548 | t1
| 45035996273730516 |
User1
45035996273704968 | public
| 45035996273793876 | foo
| 45035996273724576 |
Alice
45035996273704968 | public
| 45035996273795846 | t33
| 45035996273714428 |
Bob
(5 rows)
Administrator's Guide
Managing Users and Privileges
See Also
l
This operation immediately reassigns the sequence from the current owner to the specified new
owner.
Only the sequence owner or a superuser can change ownership, and reassignment does not
transfer grants from the original owner to the new owner; grants made by the original owner are
dropped.
Note: Renaming a table owner transfers ownership of dependent sequence objects
Administrator's Guide
Managing Users and Privileges
Example
The following example reassigns sequence ownership from the current owner to user Bob:
=> ALTER SEQUENCE sequential OWNER TO Bob;
Administrator's Guide
Managing Users and Privileges
| Sue
dbadmin | SELECT*
| Sue
dbadmin | USAGE
| Alice
dbadmin | INSERT, SELECT
| commentor
dbadmin | INSERT, SELECT
| commentor
dbadmin |
| logadmin
dbadmin |
| logadmin
dbadmin | DELETE
| logadmin
dbadmin | SELECT
| logreader
dbadmin | INSERT
| logwriter
dbadmin | USAGE
| public
dbadmin | CREATE TEMP
| public
dbadmin | USAGE
| public
dbadmin | USAGE
| public
dbadmin | USAGE
| public
dbadmin | CREATE*, CREATE TEMP*
| dbadmin
dbadmin | USAGE*, CREATE*
| dbadmin
dbadmin | INSERT*, SELECT*, UPDATE*, DELETE*, REFERENCES*
| dbadmin
dbadmin | INSERT*, SELECT*, UPDATE*, DELETE*, REFERENCES*
| dbadmin
(30 rows)
| public
| applog
| general
| public
| comments
| public
| applog
| logwriter
| logreader
| public
| applog
| public
| applog
| public
| applog
| v_internal
| vmart
| public
| v_catalog
| v_monitor
| vmart
| schema2
| public
| comments
| public
| applog
To quickly find all of the privileges that have been granted to all users on the schema named
myschema, run the following statement:
=> SELECT grantee, privileges_description FROM GRANTS
grantee | privileges_description
---------+-----------------------Bob
| USAGE, CREATE
Alice
| CREATE
(2 rows)
WHERE object_name='myschema';
Note that the vsql commands, \dp and \z, both return similar information to GRANTS:
=> \dp
Privileges
Grantee
| Grantor |
Administrator's Guide
Managing Users and Privileges
-----------+---------+-------------------------------------------------+--------+----------public
| dbadmin | USAGE
|
| v_
internal
public
| dbadmin | USAGE
|
| v_
catalog
public
| dbadmin | USAGE
|
| v_
monitor
logadmin | dbadmin |
|
|
logreader
logadmin | dbadmin |
|
|
logwriter
Fred
| dbadmin | USAGE
|
| general
Fred
| dbadmin |
|
|
logadmin
Bob
| dbadmin | USAGE
|
| general
dbadmin
| dbadmin | USAGE*, CREATE*
|
| schema2
Bob
| dbadmin | CREATE
|
| schema2
Sue
| dbadmin | USAGE
|
| general
public
| dbadmin | USAGE
|
| public
Sue
| dbadmin | USAGE
|
| public
public
| dbadmin | CREATE TEMP
|
| appdat
dbadmin
| dbadmin | CREATE*, CREATE TEMP*
|
| appdat
Sue
| dbadmin | CREATE, CREATE TEMP
|
| appdat
dbadmin
| dbadmin | INSERT*, SELECT*, UPDATE*, DELETE*, REFERENCES* | public | applog
logreader | dbadmin | SELECT
| public | applog
logwriter | dbadmin | INSERT
| public | applog
logadmin | dbadmin | DELETE
| public | applog
Sue
| dbadmin | SELECT*
| public | applog
(22 rows)
Administrator's Guide
Managing Users and Privileges
In this case, a manager can access the SSN column, while a representative is prevented from
accessing it.
=>
=>
=>
=>
=>
=>
=>
When using an access policy, you cannot use any of the following in an access expression:
n
Aggregate functions
Subquery
Analytics
UDT
If the query cannot be folded (or compressed) by the Vertica optimizer, all functions other than
SELECTare blocked. The following error message appears:
=> ERROR 0: Unable to INSERT: "Access denied due to active access policy on table
<tablename> for column (columnname>
Administrator's Guide
Managing Users and Privileges
You can create access policies only if you have dbadmin privileges.
Use the following SQL syntax to implement an access policy:
=> CREATE ACCESS POLICY ON tablename
=> FORCOLUMNname expression/column enable;
Creating an access policy depends on the expressions specified when creating the policy, and also
on the following:
l
Creating an access policy allows different users to run the same query and receive different results.
For example, you can create an access policy that a user with the role employee cannot access to
a column of bank account numbers. However, you do give access to a user with a manager role.
Conditions specified in the access policy determine whether the user can see data restricted by the
policy. This example shows how the manager role can view the entire Social Security number while
the operator role can only view the last four digits. The first five digits are masked for the operator
role (THEN SUBSTR (SSN, 8, 4)). The 8 indicates the operator sees data starting on the eighth
character (such as 123-45-6789).
=> CREATE ACCESS POLICY ON customers_table
FORCOLUMNSSN
CASE
WHENENABLED_ROLE('manager') THENSSN
WHEN ENABLED_ROLE('operator') THEN SUBSTR(SSN, 8, 4)
ELSE NULL
END
ENABLE;
Administrator's Guide
Managing Users and Privileges
Create a User
Create a Role
GRANT (Schema)
GRANT (Table)
The user enables the role with the SET ROLE statement (unless the administration user
assigned a default role to the user)
Create the access policy with the CREATE ACCESS POLICY statement.
Examples
The following examples show how to create an access policy for various situations.
Create Access Policy in Public Schema for Column in Customer Table
=> CREATE ACCESS POLICY on public.customer FOR COLUMN cid enable;
Administrator's Guide
Managing Users and Privileges
See Also
l
Grant schema or table privileges to a table non-owner to allow that user to use the access policy.
Revoke schema or table privileges to prohibit the user from using the access policy.
This example shows how to create an access policy without the user being granted privilege for the
public schema:
=> CREATE ACCESS POLICY ON public.customers_table
=> FOR COLUMN SSN
TopK projections
Aggregate projections
Pre-join projection
Administrator's Guide
Managing Users and Privileges
Sometimes, a table already has an access policy and is part of a projection. In such cases, if the
Vertica optimizer cannot fold (or compress) the query, the access query is blocked.
Administrator's Guide
Managing Users and Privileges
Role Hierarchies
You can also use roles to build hierarchies of roles; for example, you can create an administrator
role that has privileges granted non-administrator roles as well as to the privileges granted directly
to the administrator role. See also Role Hierarchy.
Roles do no supersede manually-granted privileges, so privileges directly assigned to a user are not
altered by roles. Roles just give additional privileges to the user.
Administrator's Guide
Managing Users and Privileges
Administrator's Guide
Managing Users and Privileges
PUBLIC
PSEUDOSUPERUSER
DBADMIN
DBDUSER
Predefined roles cannot be dropped or renamed. Other roles may not be granted to (or revoked from)
predefined roles except to/from PUBLIC, but predefined roles may be granted to other roles or users
or both.
Individual privileges may be granted to/revoked from predefined roles. See the SQL Reference
Manual for all of the GRANT and REVOKE statements.
DBADMIN Role
Every database has the special DBADMIN role. A superuser (or someone with the
PSEUDOSUPERUSER Role) can grant this role to or revoke this role from any user or role.
Users who enable the DBADMIN role gain these privileges:
l
Administrator's Guide
Managing Users and Privileges
You can assign additional privileges to the DBADMIN role, but you cannot assign any additional
roles; for example, the following is not allowed:
=> CREATE ROLE appviewer;
CREATE ROLE
=> GRANT appviewer TO dbadmin;
ROLLBACK 2347: Cannot alter predefined role "dbadmin"
You can, however, grant the DBADMIN role to other roles to augment a set of privileges. See Role
Hierarchy for more information.
See Also
DBADMIN User
DBDUSER Role
The special DBDUSER role must be explicitly granted by a superuser and is a predefined role. The
DBDUSER role allows non-DBADMIN users to access Database Designer using command-line
functions. Users with the DBDUSER role cannot access Database Designer using the
Administration Tools. Only DBADMIN users can run Administration Tools.
You cannot assign any additional privileges to the DBDUSER role, but you can grant the
DBDUSER role to other roles to augment a set of privileges.
Administrator's Guide
Managing Users and Privileges
Once you have been granted the DBDUSER role, you must enable it before you can run Database
Designer using command-line functions. For more information, see About Running Database
Designer Programmatically.
Important: When you create a DBADMIN user or grant the DBDUSER role, make sure to
associate a resource pool with that user to manage resources during Database Designer runs.
Multiple users can run Database Designer concurrently without interfering with each other or
using up all the cluster resources. When a user runs Database Designer, either using the
Administration Tools or programmatically, its execution is mostly contained by the user's
resource pool, but may spill over into some system resource pools for less-intensive tasks.
PSEUDOSUPERUSER Role
The special PSEUDOSUPERUSER role is automatically created in each database. A superuser
(or someone with the PSEUDOSUPERUSER role) can grant this role to another user, or revoke the
role from another user. The PSEUDOSUPERUSER cannot revoke or change any superuser
privileges.
Users with the PSEUDOSUPERUSER role enabled have all of the privileges of the database
superuser, including the ability to:
l
Create schemas
You can assign additional privileges to the PSEUDOSUPERUSER role, but you cannot assign any
additional roles; for example, the following is not allowed:
Administrator's Guide
Managing Users and Privileges
PUBLIC Role
By default, every database has the special PUBLIC role. HP Vertica grants this role to each user
automatically, and it is automatically enabled. You grant privileges to this role that every user
should have by default. You can also grant access to roles to PUBLIC, which allows any user to
access the role using the SET ROLE statement.
Note: The PUBLIC role can never be dropped, nor can it be revoked from users or roles.
Example
In the following example, if the superuser hadn't granted INSERT privileges on the table publicdata
to the PUBLIC group, the INSERT statement executed by user bob would fail:
=> CREATE TABLE publicdata (a INT, b VARCHAR);
CREATE TABLE
=> GRANT INSERT, SELECT ON publicdata TO PUBLIC;
GRANT PRIVILEGE
=> CREATE PROJECTION publicdataproj AS (SELECT * FROM publicdata);
CREATE PROJECTION
dbadmin=> \c - bob
You are now connected as user "bob".
=> INSERT INTO publicdata VALUES (10, 'Hello World');
OUTPUT
-------1
(1 row)
See Also
PUBLIC User
Administrator's Guide
Managing Users and Privileges
name
| setting
---------------+--------enabled roles |
(1 row)
A superuser can set one or more default roles for a user, which are automatically enabled at the
start of the user's session. Setting a default role is a good idea if users normally rely on the
privileges granted by one or more roles to carry out the majority of their tasks. To set a default role,
use the DEFAULT ROLE parameter of the ALTER USER statement as superuser:
=> \c vmart apps
You are now connected to database "apps" as user "dbadmin".
=> ALTER USER Bob DEFAULT ROLE logadmin;
ALTER USER
=> \c - Bob;
You are now connected as user "Bob"
=> SHOW ENABLED_ROLES;
name
| setting
---------------+---------enabled roles | logadmin
(1 row)
Notes
l
Only roles that the user already has access to can be made default.
Unlike granting a role, setting a default role or roles overwrites any previously-set defaults.
To clear any default roles for a user, use the keyword NONE as the role name in the DEFAULT
ROLE argument.
Default roles only take effect at the start of a user session. They do not affect the roles enabled
in the user's current session.
Avoid giving users default roles that have administrative or destructive privileges (the
PSEUDOSUPERUSER role or DROP privileges, for example). By forcing users to explicitly
enable these privileges, you can help prevent accidental data loss.
Administrator's Guide
Managing Users and Privileges
3. A superuser or users with administrator access to the role grant users and other roles access
to the role.
4. Users granted access to the role run the SET ROLE command to make that role active and
gain the role's privileges.
You can do steps 2 and 3 in any order. However, granting access to a role means little until the role
has privileges granted to it.
Tip: Query system tables ROLES, GRANTS, and USERS to see any directly-assigned roles.
Because these tables do not indicate whether a role is available to a user when roles could be
available through other roles (indirectly), see the HAS_ROLE() function for additional
information.
See Also
l
Role Hierarchy
In addition to granting roles to users, you can also grant roles to other roles. This lets you build
hierarchies of roles, with more privileged roles (an administrator, for example) being assigned all of
the privileges of lesser-privileged roles (a user of a particular application), in addition to the
privileges you assign to it directly. By organizing your roles this way, any privilege you add to the
application role (reading or writing to a new table, for example) is automatically made available to
the more-privileged administrator role.
Example
The following example creates two roles, assigns them privileges, then assigns them to a new
administrative role.
1. Create new table applog:
=> CREATE TABLE applog (id int, sourceID VARCHAR(32), data TIMESTAMP, event VARCHAR
(256));
Administrator's Guide
Managing Users and Privileges
6. Create a new role called logadmin, which will rule the other two roles:
=> CREATE ROLE logadmin;
8. Grant the logadmin role privileges to have the same privileges as the logreader and logwriter
roles:
=> GRANT logreader, logwriter TO logadmin;
Administrator's Guide
Managing Users and Privileges
The user Bob can now enable the logadmin role, which also includes the logreader and logwriter
roles. Note that Bob cannot enable either the logreader or logwriter role directly. A user can only
enable explicitly-granted roles.
Hierarchical roles also works with administrative access to a role:
=> GRANT logreader, logwriter TO logadmin WITH ADMIN OPTION;
GRANT ROLE
=> GRANT logadmin TO Bob;
=> \c - bob; -- connect as Bob
You are now connected as user "Bob".
=> SET ROLE logadmin; -- Enable logadmin role
SET
=> GRANT logreader TO Alice;
GRANT ROLE
Note that the user Bob only has administrative access to the logreader and logwriter roles through
the logadmin role. He doesn't have administrative access to the logadmin role, since it wasn't
granted to him with the optional WITH ADMIN OPTION argument:
=> GRANT logadmin TO Alice;
WARNING: Some roles were not granted
GRANT ROLE
For Bob to be able to grant the logadmin role, a superuser would have had to explicitly grant him
administrative access.
See Also
l
The newly-created role has no privileges assigned to it, and no users or other roles are initially
granted access to it. A superuser must grant privileges and access to the role.
Administrator's Guide
Managing Users and Privileges
Note that if any user or other role has been assigned the role you are trying to delete, the DROP
ROLE statement fails with a dependency message.
=> DROP ROLE administrator;
NOTICE: User Bob depends on Role administrator
ROLLBACK: DROP ROLE failed due to dependencies
DETAIL: Cannot drop Role administrator because other objects depend on it
HINT: Use DROP ROLE ... CASCADE to remove granted roles from the dependent users/roles
Supply the optional CASCADE parameter to drop the role and its dependencies.
=> DROP ROLE administrator CASCADE;
DROP ROLE
Example
The following example creates two roles and assigns them different privileges on a single table
called applog.
1. Create a table called applog:
=> CREATE TABLE applog (id int, sourceID VARCHAR(32), data TIMESTAMP, event VARCHAR
(256));
Administrator's Guide
Managing Users and Privileges
See the SQL Reference Manual for the different GRANT statements.
Revoking a privilege immediately affects any user sessions that have the role active. When you
revoke a privilege, it is immediately removed from users that rely on the role for the privilege.
See the SQL Reference Manual for the different REVOKE statements.
HP Vertica will return a NOTICE if you grant a role with or without admin option, to a grantee who
has already been granted that role. For example:
=> GRANT commenter to Bob;
NOTICE 4622: Role "commenter" was already granted to user "Bob"
Example
The following process illustrates how to create a role called commenter and granting user Bob
access to that role.
Administrator's Guide
Managing Users and Privileges
Based on the privileges granted to Bob by the commenter role, Bob can insert and query the
comments table.
Administrator's Guide
Managing Users and Privileges
Note that Bob does not have proper permissions to drop the table:
=> DROP TABLE comments;ROLLBACK 4000:
See Also
l
Example
To revoke access from a role, use the REVOKE (Role) statement:
1. Connect to the database as a superuser:
\c - dbadmin
Administrator's Guide
Managing Users and Privileges
Example
The following example demonstrates granting the user bob administrative access to the commenter
role, then connecting as bob and granting a role to another user.
1. Connect to the database as a superuser (or a user with administrative access):
=> \c - dbadmin
Users with administrative access to a role can also grant other users administrative access:
=> GRANT commenter TO alice WITH ADMIN OPTION;
GRANT ROLE
As with all user privilege models, database superusers should be cautious when granting any user a
role with administrative privileges. For example, if the database superuser grants two users a role
with administrative privileges, both users can revoke the role of the other user. This example shows
granting the appadmin role (with administrative privileges) to users bob and alice. After each user
has been granted the appadmin role, either use can connect as the other will full privileges.
Administrator's Guide
Managing Users and Privileges
Example
The following example demonstrates revoking administrative access from Alice for the commenter
role.
1. Connect to the database as a superuser (or a user with administrative access)
\c - dbadmin
Enabling Roles
By default, roles aren't enabled automatically for a user account. (See Default Roles for Database
Users for a way to make roles enabled automatically.) Users must explicitly enable a role using the
SET ROLE statement. When users enable a role in their session, they gain all of the privileges
assigned to that role. Enabling a role does not affect any other roles that the users have active in
their sessions. They can have multiple roles enabled simultaneously, gaining the combined
privileges of all the roles they have enabled, plus any of the privileges that have been granted to
them directly.
=> SELECT * FROM applog;
ERROR: permission denied for relation applog
=> SET ROLE logreader;
SET
Administrator's Guide
Managing Users and Privileges
You can enable all of the roles available to your user account using the SET ROLE ALL statement.
=> SET ROLE ALL;SET
=> SHOW ENABLED_ROLES;
name
|
setting
---------------+-----------------------------enabled roles | logreader, logwriter
(1 row)
See Also
l
Disabling Roles
To disable all roles, use the SET ROLE NONE statement:
=> SET ROLE NONE;SET
=> SHOW ENABLED_ROLES;
name
| setting
---------------+--------enabled roles |
(1 row)
You can find the roles available to your account using the SHOW AVAILABLE ROLES statement:
Administrator's Guide
Managing Users and Privileges
Note: An asterisk (*) in the output means that role was granted WITH ADMIN OPTION.
In this example, a superuser wants to verify that the logadmin role has been granted to user Ted:
Administrator's Guide
Managing Users and Privileges
The output returns boolean value t for true, denoting that Ted is assigned the specified logadmin
role:
HAS_ROLE
---------t
(1 row)
Note that if a superuser omits the user_name argument, the function looks up that superuser's role.
The following output indicates that this superuser is not assigned the logadmin role:
dbadmin=> SELECT HAS_ROLE('logadmin');
HAS_ROLE
---------f
(1 row)
Output of the function call with user Alice indicates that she is not granted the logadmin role:
dbadmin=> SELECT HAS_ROLE('Alice', 'logadmin');
HAS_ROLE
---------f
(1 row)
To view additional information about users, roles and grants, you can also query the following
system tables in the V_CATALOG schema to show directly-assigned roles:
l
ROLES
GRANTS
USERS
Note that the system tables do not indicate whether a role is available to a user when roles could be
available through other roles (indirectly). You need to call the HAS_ROLE() function for that
information.
Users
This command returns all columns from the USERS system table:
=> SELECT * FROM users;
Administrator's Guide
Managing Users and Privileges
-[RECORD 1 ]
------------------+--------------------------user_id
| 45035996273704962
user_name
| dbadmin
is_super_user
| t
profile_name
| default
is_locked
| f
lock_time
|
resource_pool
| general
memory_cap_kb
| unlimited
temp_space_cap_kb | unlimited
run_time_cap
| unlimited
all_roles
| dbadmin*, pseudosuperuser*default_roles
| dbadmin*, pseudosuperuser*
Note: An asterisk (*) in table output for all_roles and default_roles columns indicates a role
granted WITH ADMIN OPTION.
Roles
The following command returns all columns from the ROLES system table:
=> SELECT * FROM roles;
role id
|
name
| assigned_roles
-------------------+-----------------+------------------45035996273704964 | public
|
45035996273704964 | dbduser
|
45035996273704964 | dbadmin
| dbduser*
45035996273704964 | pseudosuperuser | dbadmin*
Grants
The following command returns all columns from the GRANTS system table:
=> SELECT * FROM grants;
grantor | privileges_description | object_schema | object_name | grantee
---------+------------------------+---------------+-------------+--------dbadmin | USAGE
|
| public
| public
dbadmin | USAGE
|
| v_internal | public
dbadmin | USAGE
|
| v_catalog
| public
dbadmin | USAGE
|
| v_monitor
| public
(4 rows)
Administrator's Guide
Managing Users and Privileges
Administrator's Guide
Managing Users and Privileges
MC Privilege Groups
l
MC configuration privileges are made up of roles that control what users can configure on the
MC, such as modify MC settings, create/import HP Vertica databases, restart MC, create an
HP Vertica cluster through the MC interfac, and create and manage MC users.
MC database privileges are made up of roles that control what users can see or do on an MCmanaged HP Vertica database, such as view the database cluster state, query and session
activity, monitor database messages and read log files, replace cluster nodes, and stop
databases.
MC database privileges cannot alter or override the HP Vertica database user's privileges and
roles. MC users who are granted a database role inherit only the privileges assigned to the database
user account to which the MC user is mapped. To provide maximum access, use the dbadmin
username and password.
For more information on MC user/database user association, see Mapping an MC User to a
Database User's Privileges.
MC Configuration Privileges
MC Database Privileges
MC
administrators
(SUPER and
ADMIN)
accounts.
Administrator's Guide
Managing Users and Privileges
User Type
MC Configuration Privileges
MC Database Privileges
IT users (IT)
HP Vertica
OnDemand
users
(MANAGER
and
ASSOCIATE)
MCconfiguration role.
Perform no administrative
Database
assigned databases.
Described in
Described in MC Configuration
About MC
Privileges
Users
See Also
l
About MC Users
Creating an MC User
MC Configuration Privileges
When you create an MC user, you assign an MC configuration access level (role). For the most
part, MC configuration privileges control a user's ability to create users and manage MC settings on
Administrator's Guide
Managing Users and Privileges
the MC interface.
MC.
IT Role (mc)Full access to all MC functionality. Database access depends on the mapped
database user.
MANAGER role (mc)Full access to all MC functionality. Database access depends on the
user.
You grant MC configuration privileges at the same time you create the user's account, using the
MC Settings page. You can also change MC access levels using this page. See Creating an MC
User for details.
You can also grant non-administrators access to one or more databases managed by MC. See MC
Database Privileges for details.
ADMIN
Configure MC settings:
Yes
IT
MANAGER
NONE
Administrator's Guide
Managing Users and Privileges
MC Access Privileges
ADMIN
Yes
IT
MANAGER
NONE
clusters
l
Yes
Yes
databases
Enable or disable user access to MC
Yes
Yes
Yes
interface
Monitor user activity on MC
Yes
Yes
configured state
Restart Management Console
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
alerts
See Also
l
About MC Users
Administrator's Guide
Managing Users and Privileges
MC Database Privileges
Creating an MC User
Oversees the entire Management Console, including all MC-managed database clusters
Note: This user inherits the privileges/roles of the user name supplied when importing an
HP Vertica database into MC. HP recommends that you use the database administrator's
credentials.
Creates the first MC user accounts and assigns them an MC configuration role
The MC super administrator account is unique. Unlike other MC users you create, including other
MC administrators, the MC super account cannot be altered or dropped, and you cannot grant the
SUPER role to other MC users. The only property you can change for the MC super is the
password. Otherwise the SUPER role has the same privileges on MC as the ADMIN Role (mc).
On MC-managed HP Vertica databases, SUPER has the same privileges as ADMIN Role (db).
The MC super account does not exist within the LDAP server. This account is also different from
the special dbadmin account that gets created during an HP Vertica installation, whose privileges
are governed by the DBADMIN Role. The HP Vertica-created dbadmin is a Linux account that
owns the database catalog and storage locations and can bypass database authorization rules,
such as creating or dropping schemas, roles, and users. The MC super does not have the same
privileges as dbadmin.
Administrator's Guide
Managing Users and Privileges
See Also
l
Configuring MC
Creating an MC User
Managing MC Users
Modify MC settings, such as storage locations and ports, restart the MC process, and reset MC
to its original, unconfigured state
Administrator's Guide
Managing Users and Privileges
Add new users and map them to one or more HP Vertica databases by granting an MC
database-level role
statements.
Administrator's Guide
Managing Users and Privileges
See Also
l
Creating an MC User
Managing MC Users
IT Role (mc)
MC IT users can monitor all MC-managed databases, view MC-level (non database) messages,
logs, and alerts, disable or enable user access to MC, and reset non-LDAP user passwords. You
can also assign MC IT users specific database privileges, which you do by mapping IT users to a
user on a database. In this way, the MC IT user inherits the privileges assigned to the database
user to which he/she is mapped.
MC database IT role
Monitor MC-managed
settings.
Can also be mapped to one or more HP Vertica databases.
Administrator's Guide
Managing Users and Privileges
See Also
l
IT Role (db)
The Manager role has similar configuration privileges to the IT configuration role. Unlike IT users,
Managers can also create, edit, and delete users in User Settings.
Administrator's Guide
Managing Users and Privileges
See Also
l
About MC Users
MC Database Privileges
IT Role (db)
MC Database Privileges
When you create MC users, you first assign them MC configuration privileges, which controls what
they can do in MC. In the same user-creation operation, you grant access to one or more databases
managed by MC. MC database access does not give the MC user privileges directly on HP Vertica;
it provides MC users varying levels of access to assigned database functionality through the MC
interface.
Assign users an MC database level through one of the following roles:
l
ADMIN Role (db)Full access to all MC-managed databases. Actual privileges Admins inherit
depend on those defined for the database user account that is used to create or import the HP
Vertica database into MC.
Administrator's Guide
Managing Users and Privileges
ASSOCIATErole (db)Full access to all databases managed by MC. Cannot start, stop, or
drop a database. Actual privileges that Associates receive depend on those defined for the
database user account to which the Associate user is mapped.
IT Role (db)Can start and stop a database but cannot remove it from the MC interface or drop
it.
USER Role (db)Can only view database information through the database Overview and
Activities pages. Viewing more detailed data is restricted.
ADMIN
ASSOCIATE
IT
USER
View messages
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Start a database
Yes
Stop a node
Yes
Yes
Yes
Yes
View MC settings
Yes
Yes
Yes
read/unread
The following privileges are governed by the HP Vertica database user account:
Database-Specific Privileges
ADMIN
ASSOCIATE
Yes
Yes
Yes
Yes
Administrator's Guide
Managing Users and Privileges
Database-Specific Privileges
ADMIN
ASSOCIATE
Yes
Yes
Yes
Yes
Stop a database
Yes
Rebalance a database
Yes
Drop a database
Yes
Yes
Yes
Yes
Prevent the MC user from doing or seeing anything that is not allowed by the privileges set up for
the user account on the server database
Privileges assigned to the database user supersede privileges of the MC user if there is a conflict,
such as stopping a database. When the MC user logs in to MC using an MC user name and
password, HP Vertica compares privileges for database-related activities to the privileges on the
database account to which you mapped the MC user. Only when the user has both MC privileges
and corresponding database privileges does HP Vertica expose operations in MC to that user.
Tip: As a best practice, you should identify, in advance, the appropriate HP Vertica database
user account that has privileges or roles similar to one of the MC database roles.
See Creating an MC User and Mapping an MC User to a Database User's Privileges for more
information.
See Also
l
About MC Users
Administrator's Guide
Managing Users and Privileges
MC Configuration Privileges
The following MC-managed database operations depend on the database user's role that you
mapped this ADMIN user to:
l
Administrator's Guide
Managing Users and Privileges
Note: Database access granted through Management Console never overrides roles granted
on a specific HP Vertica database.
statements.
See Also
l
Administrator's Guide
Managing Users and Privileges
IT Role (db)
IT can view most details about an MC-managed database, such as messages (and mark them
read/unread), the database overall health and activity/resources, cluster and node state, and MC
settings. You grant and manage user role assignments through the MC Settings > User
management page on the MC.
MC configuration IT
Monitor MC-managed
settings.
access.
See Also
l
IT Role (mc)
Administrator's Guide
Managing Users and Privileges
See Also
l
MC Configuration Privileges
Prerequisites
Before you grant database access to an MC user, make sure you have read the prerequisites in
Creating an MC User.
Administrator's Guide
Managing Users and Privileges
4. Next to the DB access levels section, click Add and provide the following database access
credentials:
i. Choose a database. Select a database from the list of MC-discovered (databases
that were created on or imported into the MC interface).
ii. Database username. Enter an existing database user name or, if the database is
running, click the ellipses [] to browse for a list of database users, and select a
name from the list.
iii. Database password. Enter the password to the database user account (not this
username's password).
iv. Restricted access. Chose a database level (ADMIN, IT, or USER) for this user.
v. Click OK to close the Add permissions dialog box.
5. Optionally change the user's Status (enabled is the default).
6. Click Save.
See Mapping an MC User to a Database User's Privileges for a graphical illustration of how easy it
is to map the two user accounts.
See Also
l
About MC Users
Creating an MC User
Administrator's Guide
Managing Users and Privileges
To give alice database privileges, click to highlight her MC username, click Edit, and the Edit
existing user page displays with no resources (databases) assigned to MC user alice.
Administrator's Guide
Managing Users and Privileges
Click Add, and when the Add permissions dialog box opens, choose a database from the menu.
In the same Add permissions dialog box, after you select a database, you need to enter the user
name of the database user account that you want to map alice to. To see a list of database user
names, click the ellipses [] and select a name from the list. In this example, you already know
that database user carol has privileges to stop and start the database, but the alice database
Administrator's Guide
Managing Users and Privileges
account can only view certain tables. On MC, you want alice to have similar privileges to carol, so
you map MC alice to database carol.
After you click OK, remember to assign MC user alice an MC database level. In this case, choose
IT, a role that has permissions to start and stop the selected database.
Administrator's Guide
Managing Users and Privileges
Enter the database password, click OK , close the confirmation dialog box, and click Save.
That's it.
Administrator's Guide
Managing Users and Privileges
See Granting Database Access to MC Users and Mapping an MC User to a Database User's
Privileges for details.
Administrator's Guide
Managing Users and Privileges
To modify that information, highlight a user and click Edit. In the dialog box that opens (shown in
example below), Bob's role on the mcdb database is ADMIN. To change Bob's role from this dialog
box, click Edit and assign Bob a different database-access role.
From the User management option, select each user and grant database access, one user at a
time
This procedure describes how to add several users to one database at once. If you want to add
users one at a time, see Creating an MC User.
Administrator's Guide
Managing Users and Privileges
3. To add new users, click Add and select the MC username you want to add to the database
from the drop-down list.
4. Enter an existing Database username on the selected database or click the ellipses button
[] to browse for names. (This is the database account you want to map the selected user to.)
5. Enter the database password (not this username's password).
6. Choose a database-access role (ADMIN, IT, ASSOCIATE or USER) for this user.
7. Click OK to close the Add access to resource dialog box.
8. Perform steps 3-7 for each user you want to add to the selected database, and then click Save.
See Also
l
About MC Users
MC Mapping Matrix
The following table shows the three different MC configuration roles, ADMIN, IT, and NONE,
combined with the type of privileges a user granted that role inherits when mapped to a specific
database-level role.
MC
MC
configuration database
level
level
The combination lets the user ...
ADMIN Role
All
(mc)
(implicit)
Administrator's Guide
Managing Users and Privileges
MC
MC
configuration database
level
level
The combination lets the user ...
IT Role (mc)
ADMIN
Role (db)
IT Role (mc)
IT Role
(db)
IT Role (mc)
USER
Role (db)
Administrator's Guide
Managing Users and Privileges
MC
MC
configuration database
level
level
The combination lets the user ...
NONE Role
ADMIN
(mc)
Role (db)
NONE Role
IT Role
(mc)
(db)
NONE Role
USER
(mc)
Role (db)
Administrator's Guide
Using the Administration Tools
toolname
options
-h--help
-a--help_all
If you omit toolname and options parameters, the Main Menu dialog box appears inside your
console or terminal window with a dark blue background and a title on top. The screen captures
used in this documentation set are cropped down to the dialog box itself, as shown below.
Administrator's Guide
Using the Administration Tools
If you are unfamiliar with this type of interface, read Using the Administration Tools Interface before
you do anything else.
Between Dialogs
While the Administration Tools are working, you see the command line processing in a window
similar to the one shown below. Do not interrupt the processing.
Administrator's Guide
Using the Administration Tools
Enter [Return]
In all dialogs, when you are ready to run a command, select a file, or cancel the dialog, press the
Enter key. The command descriptions in this section do not explicitly instruct you to press Enter.
OK - Cancel - Help
The OK, Cancel, and Help buttons are
present on virtually all dialogs. Use the
tab, space bar, or right and left arrow
keys to select an option and then press
Enter. The same keystrokes apply to
dialogs that present a choice of Yes or
No.
Administrator's Guide
Using the Administration Tools
Menu Dialogs
Some dialogs
require that you
choose one
command from
a menu. Type
the
alphanumeric
character
shown or use
the up and
down arrow
keys to select a
command and
then press
Enter.
List Dialogs
In a list dialog, use the up and down arrow keys
to highlight items, then use the space bar to
select the items (which marks them with an X).
Some list dialogs allow you to select multiple
items. When you have finished selecting items,
press Enter.
Form Dialogs
In a form dialog (also referred to as a dialog box), use the tab key to cycle between OK, Cancel,
Help, and the form field area. Once the cursor is in the form field area, use the up and down arrow
keys to select an individual field (highlighted) and enter information. When you have finished
entering information in all fields, press Enter.
Administrator's Guide
Using the Administration Tools
Help Buttons
Online help is provided in the form of text dialogs. If you have trouble viewing the help, see Notes
for Remote Terminal Users in this document.
Connect to database
Stop database
Replace node (assuming node that is down is the one being replaced)
Administrator's Guide
Using the Administration Tools
The following operations work with unresponsive nodes; however, you might have to repeat the
operation on the failed nodes after they are back in operation:
l
The following management functions within the Administration Tools require that all nodes be UP in
order to be operational:
l
Create database
Drop database
Administrator's Guide
Using the Administration Tools
Most of the online help in the Administration Tools is context-sensitive. For example, if you use
up/down arrows to select a command, press tab to move to the Help button, and press return, you
get help on the selected command.
Administrator's Guide
Using the Administration Tools
In a Menu Dialog
1. Use the up and down arrow keys to choose the command for which you want help.
2. Use the Tab key to move the cursor to the Help button.
3. Press Enter (Return).
In a Dialog Box
1. Use the up and down arrow keys to choose the field on which you want help.
2. Use the Tab key to move the cursor to the Help button.
3. Press Enter (Return).
Administrator's Guide
Using the Administration Tools
Scrolling
Some help files are too long for a single screen. Use the up and down arrow keys to scroll through
the text.
Password Authentication
When you create a new user with the CREATE USER command, you can configure the password
or leave it empty. You cannot bypass the password if the user was created with a password
configured. You can change a user's password using the ALTER USER command.
See Implementing Security for more information about controlling database authorization through
passwords.
Tip: Unless the database is used solely for evaluation purposes, HP recommends that all
database users have encrypted passwords.
When you restore the node to the database cluster, you can use the Administration Tools to update
the node with the latest Administration Tools metadata:
1. Log on to a host that contains the metadata you want to transfer and start the Administration
Tools. (See Using the Administration Tools.)
2. On the Main Menu in the Administration Tools, select Configuration Menu and click OK.
3. On the Configuration Menu, select Distribute Config Files and click OK.
Administrator's Guide
Using the Administration Tools
HP Vertica Functionality
Management
Console
Administration
Tools
Yes
No
Yes
No
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
View databases
Yes
Yes
Yes
No
Drop a database
Yes
Yes
Vertica
Manage/monitor one or more databases and clusters
through a UI
Administrator's Guide
Using the Administration Tools
HP Vertica Functionality
Management
Console
Administration
Tools
Yes
Yes
Yes
Yes
No
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
No
Yes
No
Yes
No
Yes
No
Designer)
Modify a physical schema design (Database
Designer)
resource usage
information
Use LDAP to authenticate users with company
credentials
database
Enable and disable user access to MC and/or the
database
Audit user activity on database
Administrator's Guide
Using the Administration Tools
HP Vertica Functionality
Management
Console
Administration
Tools
Yes
No
Yes
No
Management Console Provides some, but Not All of the Functionality Provided By the
Administration Tools. MC Also Provides Functionality Not Available in the Administration Tools.
See Also
l
Administrator's Guide
Using the Administration Tools
2. If some hosts are UP and some DOWN, restart the specific host that is down using Restart
HP Vertica on Host from the Administration Tools, or you can start the database as described
in Starting and Stopping the Database (unless you have a known node failure and want to
continue in that state.)
Administrator's Guide
Using the Administration Tools
See Also
l
When you create a new user with the CREATE USER command, you can configure the
password or leave it empty. You cannot bypass the password if the user was created with a
password configured. You can change a user's password using the ALTER USER command.
The Administration Tools connect to the database and transfer control to vsql.
Administrator's Guide
Using the Administration Tools
See Also
l
CREATE USER
ALTER USER
Administrator's Guide
Using the Administration Tools
2. Open the Administration Tools. See Using the Administration Tools for information about
accessing the Administration Tools.
3. On the Main Menu, select Start Database,and then select OK.
4. Select the database to start, and then click OK.
Caution: HP strongly recommends that you start only one database at a time. If you start
more than one database at any time, the results are unpredictable. Users could encounter
resource conflicts or perform operations in the wrong database.
Administrator's Guide
Using the Administration Tools
Stopping a Database
To stop a running database, take these steps:
1. Use View Database Cluster State to make sure that all nodes are up. If all nodes are not up,
see Restarting HP Vertica on Host.
2. On the Main Menu, select Stop Database, and click OK.
3. Select the database you want to stop, and click OK.
4. Enter the password if asked, and click OK.
5. A message confirms that the database has been successfully stopped. Click OK.
Error
If users are connected during shutdown operations, you cannot stop a database. The
Administration Tools displays a message similar to the following:
Uable to shutdown database VMart.
Error:NOTICE 2519:
Description
The message indicates that there are active user connections (sessions). For example, Database
Designer may be building or deploying a design. See Managing Sessions in the Administrator's
Guide for more information.
Resolution
The following examples were taken from a different database.
1. To see which users are connected, connect to the database and query the SESSIONS system
table described in the SQL Reference Manual. For example:
Administrator's Guide
Using the Administration Tools
-[RECORD 1 ]
node_name
| site01
user_name
| dbadmin
client_hostname
| 127.0.0.1:57141
login_timestamp
| 2009-06-07 14:41:26
session_id
| rhel5-1-30361:0xd7e3e:994462853
transaction_start
| 2009-06-07 14:48:54
transaction_id
| 45035996273741092
transaction_description | user dbadmin (select * from session;)
statement_start
| 2009-06-07 14:53:31
statement_id
| 0
last_statement_duration | 1
current_statement
| select * from sessions;
ssl_state
| None
authentication_method
| Trust
-[RECORD 2 ]
node_name
| site01
user_name
| dbadmin
client_hostname
| 127.0.0.1:57142
login_timestamp
| 2009-06-07 14:52:55
session_id
| rhel5-1-30361:0xd83ac:1017578618
transaction_start
| 2009-06-07 14:53:26
transaction_id
| 45035996273741096
transaction_description | user dbadmin (COPY ClickStream_Fact FROM
'/data/clickstream/1g/ClickStream_Fact.tbl' DELIMITER '|' NULL '\\n' DIRECT;)
statement_start
| 2009-06-07 14:53:26
statement_id
| 17179869528
last_statement_duration | 0
current_statement
| COPY ClickStream_Fact FROM
'/data/clickstream/1g/ClickStream_Fact.tbl' DELIMITER '|' NULL '\\n' DIRECT;
ssl_state
| None
authentication_method
| Trust
The current statement column of Record 1 shows that session is the one you are using to query the
system table. Record 2 shows the session that must end before the database can be shut down.
1. If a statement is running in a session, that session must be closed. Use the function CLOSE_
SESSION or CLOSE_ALL_SESSIONS described in the SQL Reference Manual.
-[RECORD 1 ]
| site01
| dbadmin
Administrator's Guide
Using the Administration Tools
client_hostname
client_pid
login_timestamp
session_id
client_label
transaction_start
transaction_id
transaction_description
statement_start
statement_id
last_statement_duration_us
current_statement
ssl_state
authentication_method
-[RECORD 2 ]
node_name
user_name
client_hostname
client_pid
login_timestamp
session_id
client_label
transaction_start
transaction_id
transaction_description
|
|
|
|
|
|
|
|
|
|
|
|
|
|
127.0.0.1:57141
17838
2009-06-07 14:41:26
rhel5-1-30361:0xd7e3e:994462853
|
|
|
|
|
|
|
|
|
|
site01
dbadmin
127.0.0.1:57142
17839
2009-06-07 14:52:55
rhel5-1-30361:0xd83ac:1017578618
2009-06-07 14:48:54
45035996273741092
user dbadmin (select * from sessions;)
2009-06-07 14:53:31
0
1
select * from sessions;
None
Trust
2009-06-07 14:53:26
45035996273741096
user dbadmin (COPY ClickStream_Fact FROM
'/data/clickstream/1g/ClickStream_Fact.tbl'
DELIMITER '|' NULL '\\n' DIRECT;)
statement_start
| 2009-06-07 14:53:26
statement_id
| 17179869528
last_statement_duration_us | 0
current_statement
| COPY ClickStream_Fact FROM
'/data/clickstream/1g/ClickStream_Fact.tbl'
DELIMITER '|' NULL '\\n' DIRECT;
ssl_state
| None
authentication_method
| Trust
=> SELECT CLOSE_SESSION('rhel5-1-30361:0xd83ac:1017578618');
-[RECORD 1 ]
close_session | Session close command sent. Check sessions for progress.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
-[RECORD 1 ]
site01
dbadmin
127.0.0.1:57141
17838
2009-06-07 14:41:26
rhel5-1-30361:0xd7e3e:994462853
2009-06-07 14:48:54
45035996273741092
user dbadmin (select * from sessions;)
2009-06-07 14:54:11
0
98
select * from sessions;
Administrator's Guide
Using the Administration Tools
ssl_state
authentication_method
| None
| Trust
2. Query the SESSIONS table again. For example, two columns have changed:
n
stmt_duration now indicates how long the statement ran in milliseconds before being
interrupted.
The SELECT statements that call these functions return when the interrupt or close message
has been delivered to all nodes, not after the interrupt or close has completed.
3. Query the SESSIONS table again. When the session no longer appears in the SESSION table,
disconnect and run the Stop Database command.
Controlling Sessions
The database administrator must be able to disallow new incoming connections in order to shut
down the database. On a busy system, database shutdown is prevented if new sessions connect
after the CLOSE_SESSION or CLOSE_ALL_SESSIONS() command is invokedand before the
database actually shuts down.
One option is for the administrator to issue the SHUTDOWN('true') command, which forces the
database to shut down and disallow new connections. See SHUTDOWN in the SQL Reference
Manual.
Another option is to modify the MaxClientSessions parameter from its original value to 0, in order
to prevent new non-dbadmin users from connecting to the database.
1. Determine the original value for the MaxClientSessions parameter by querying the V_
MONITOR.CONFIGURATIONS_PARAMETERS system table:
=> SELECT CURRENT_VALUE FROM CONFIGURATION_PARAMETERS WHERE parameter_
name='MaxClientSessions';
CURRENT_VALUE
--------------50
(1 row)
Administrator's Guide
Using the Administration Tools
When the session no longer appears in the SESSIONS table, disconnect and run the Stop
Database command.
5. Restart the database.
6. Restore the MaxClientSessions parameter to its original value:
=> ALTERDATABASE mydb SET MaxClientSessions = 50;
Notes
You cannot stop databases if your password has expired. The Administration Tools displays an
error message if you attempt to do so. You need to change your expired password using vsql before
you can shut down a database.
Administrator's Guide
Using the Administration Tools
4. Select the Host that you want to restart, and click OK.
5. Select View Database Cluster State again to make sure that all nodes are up.
Administrator's Guide
Using the Administration Tools
1. On the Main Menu, click Configuration Menu, and then click OK.
Creating a Database
1. On the Configuration Menu, click Create Database and then click OK.
2. Enter the name of the database and an optional comment. Click OK.
3. Enter a password.
If you do not enter a password, you are prompted to indicate whether you want to enter a
password. Click Yes to enter a password or No to create a database without a superuser
password.
Caution: If you do not enter a password at this point, superuser password is set to empty.
Unless the database is for evaluation or academic purposes, HP strongly recommends
that you enter a superuser password.
Administrator's Guide
Using the Administration Tools
5. Select the hosts to include in the database. The hosts in this list are the ones that were
specified at installation time (install_vertica -s).
6. Specify the directories in which to store the catalog and data files.
Note: Catalog and data paths must contain only alphanumeric characters and cannot
have leading space characters. Failure to comply with these restrictions could result in
database creation failure.
Note: : Do not use a shared directory for more than one node. Data and catalog directories
must be distinct for each node. Multiple nodes must not be allowed to write to the same
data or catalog directory.
7. Check the current database definition for correctness, and click Yes to proceed.
Administrator's Guide
Using the Administration Tools
8. A message indicates that you have successfully created a database. Click OK.
Dropping a Database
This tool drops an existing database. Only the Database Administrator is allowed to drop a
database.
1. Stop the database as described in Stopping a Database.
2. On the Configuration Menu, click Drop Database and then click OK.
3. Select the database to drop and click OK.
4. Click Yes to confirm that you want to drop the database.
5. Type yes and click OK to reconfirm that you really want to drop the database.
6. A message indicates that you have successfully dropped the database. Click OK.
Notes
In addition to dropping the database, HP Vertica automatically drops the node definitions that refer
to the database unless:
l
Another database uses a node definition. If another database refers to any of these node
definitions, none of the node definitions are dropped.
A node definition is the only node defined for the host. (HP Vertica uses node definitions to
locate hosts that are available for database creation, so removing the only node defined for a
host would make the host unavailable for new databases.)
Administrator's Guide
Using the Administration Tools
Viewing a Database
This tool displays the characteristics of an existing database.
1. On the Configuration Menu, select View Database and click OK.
2. Select the database to view.
The name and location of the log file for the database.
Administrator's Guide
Using the Administration Tools
K-Safe Nodes are automatically restarted if the database cluster is still UP. This is the
default setting.
Note: Always does not work if a single node database was not shutdown cleanly or
crashed.
6. Click OK.
Note: You do not need to initialize RAID recover in runlevel 1; simply validate that it can
Administrator's Guide
Using the Administration Tools
recover.
3. Once the hardware is confirmed consistent, only then reboot to runlevel 3 or higher.
At this point, the network activates, and HP Vertica rejoins the cluster and automatically recovers
any missing data. Note that, on a single-node database, if any files that were associated with a
projection have been deleted or corrupted, HP Vertica will delete all files associated with that
projection, which could result in data loss.
2. On the AdminTools Main Menu, click Configuration Menu, and then click OK.
3. On the Configuration Menu, click Install External Procedure and then click OK.
4. Select the database on which you want to install the external procedure.
5. Either select the file to install or manually type the complete file path, and then click OK.
6. If you are not the superuser, you are prompted to enter your password and click OK.
Administrator's Guide
Using the Administration Tools
Administrator's Guide
Using the Administration Tools
Important note:
In HP Vertica 4.1, the default for the HistoryRetentionTime configuration parameter changed to
0, which means that HP Vertica only keeps historical data when nodes are down. This new setting
effectively prevents the use of the Administration Tools 'Roll Back Database to Last Good
Administrator's Guide
Using the Administration Tools
Epoch' option because the AHM remains close to the current epoch and a rollback is not permitted
to an epoch prior to the AHM. If you rely on the Roll Back option to remove recently loaded data,
consider setting a day-wide window for removing loaded data; for example:
=> ALTERDATABASE mydb SET HistoryRetentionTime = 86400;
1. On the Advanced Menu, select Stop HP Vertica on Host and click OK.
2. Select the hosts to stop.
If the command succeeds View Database Cluster State shows that the selected hosts are
DOWN.
Administrator's Guide
Using the Administration Tools
If the command fails to stop any selected nodes, proceed to Killing HP Vertica Process on
Host.
1. On the Advanced menu, select Kill HP Vertica Process on Host and click OK.
2. Select the hosts on which to kills the HP Vertica process.
Administrator's Guide
Using the Administration Tools
4. If the command succeeds, View Database Cluster State shows that the selected hosts are
DOWN.
Note: If you are using HP Vertica Community Edition, follow the instructions in HP Vertica
License Renewals or Upgrades for instructions to upgrade to an HP Vertica Enterprise Edition
or evaluation license key.
Managing Clusters
Cluster Management lets you add, replace, or remove hosts from a database cluster. These
processes are usually part of a larger process of adding, removing, or replacing a database node.
Administrator's Guide
Using the Administration Tools
Note: View the database state to verify that it is running. See View Database Cluster State. If
the database isn't running, restart it. See Starting the Database.
Administrator's Guide
Using the Administration Tools
During installation
HP Vertica immediately converts any hostname you provide through command line options -hosts, -add-hosts or --remove-hosts to its IP address equivalent.
n
If you provide a hostname during installation that resolves to multiple IP addresses (such as
in multi-homed systems), the installer prompts you to choose one IP address.
HP Vertica retains the name you give for messages and prompts only; internally it stores
these hostnames as IP addresses.
Metadata is more portable because HP Vertica does not require the names of the hosts in the
cluster to be exactly the same when you install or upgrade your database.
Syntax
> /opt/vertica/bin/admintools [-t | --tool ] toolname [options ]
Note: For convenience, you can add /opt/vertica/bin to your search path.
Administrator's Guide
Using the Administration Tools
Parameters
[--tool | -t ]
toolname
[ options ]
-h--help
-a--help_all
Tools
To return a description of the tools you can access, issue the following command at a command
prompt:
$admintools -a
Usage:
adminTools [-t | --tool] toolName [options]
Valid tools are:
command_host
config_nodes
connect_db
create_db
database_parameters
db_add_node
db_remove_node
db_replace_node
db_status
distribute_config_files
drop_db
host_to_node
install_package
install_procedure
kill_host
kill_node
license_audit
list_allnodes
list_db
list_host
list_node
list_packages
Administrator's Guide
Using the Administration Tools
logrotate
node_map
rebalance_data
restart_db
restart_node
return_epoch
set_restart_policy
show_active_db
start_db
stop_db
stop_host
stop_node
uninstall_package
upgrade_license_key
view_cluster
------------------------------------------------------------------------Usage: command_host [options]
Options:
-h, --help
-c CMD, --command=CMD
Command to run
------------------------------------------------------------------------Usage: config_nodes [options]
Options:
-h, --help
-f NODEHOSTFILE, --file=NODEHOSTFILE
File containing list of nodes, hostnames, catalog
path, and datapath (node<whitespace>host<whitespace>ca
talogPath<whitespace>dataPath one per line)
-c, --check
-s SKIPANALYZENODE, --skipanalyzenode=SKIPANALYZENODE
skipanalyzenode
------------------------------------------------------------------------Usage: connect_db [options]
Options:
-h, --help
-d DB, --database=DB
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
------------------------------------------------------------------------Usage: create_db [options]
Options:
-h, --help
Administrator's Guide
Using the Administration Tools
-s NODES, --hosts=NODES
comma-separated list of hosts to participate in
database
-d DB, --database=DB
-c CATALOG, --catalog_path=CATALOG
Path of catalog directory[optional] if not using
compat21
-D DATA, --data_path=DATA
Path of data directory[optional] if not using compat21
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes [optional]
-l LICENSEFILE, --license=LICENSEFILE
Database license [optional]
-P POLICY, --policy=POLICY
Database restart policy [optional]
--compat21
hostnames
------------------------------------------------------------------------Usage: database_parameters [options]
Options:
-h, --help
-d DB, --database=DB
Name of database
-P PARAMETER, --parameter=PARAMETER
Database parameter
-c COMPONENT, --component=COMPONENT
Component[optional]
-s SUBCOMPONENT, --subcomponent=SUBCOMPONENT
Sub Component[optional]
-p PASSWORD, --password=PASSWORD
Database password[optional]
------------------------------------------------------------------------Usage: db_add_node [options]
Options:
-h, --help
-d DB, --database=DB
-s HOSTS, --hosts=HOSTS
Comma separated list of hosts to add to database
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
Administrator's Guide
Using the Administration Tools
-a AHOSTS, --add=AHOSTS
Comma separated list of hosts to add to database
-i, --noprompts
--compat21
hostnames
------------------------------------------------------------------------Usage: db_remove_node [options]
Options:
-h, --help
-d DB, --database=DB
-s HOSTS, --hosts=HOSTS
Name of the host to remove from the db
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-i, --noprompts
--compat21
hostnames
------------------------------------------------------------------------Usage: db_replace_node [options]
Options:
-h, --help
-d DB, --database=DB
-o ORIGINAL, --original=ORIGINAL
Name of host you wish to replace
-n NEWHOST, --new=NEWHOST
Name of the replacement host
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-i, --noprompts
-s STATUS, --status=STATUS
Database status UP,DOWN or ALL(list running dbs UP,list down dbs - DOWN list all dbs - ALL
------------------------------------------------------------------------Usage: distribute_config_files
Sends admintools.conf from local host to all other hosts in the cluster
Administrator's Guide
Using the Administration Tools
-d DB, --database=DB
Database to be dropped
-s HOST, --host=HOST
-d DBNAME, --dbname=DBNAME
database name
-p PASSWORD, --password=PASSWORD
database admin password
-P PACKAGE, --package=PACKAGE
specify package or 'all' or 'default'
------------------------------------------------------------------------Usage: install_procedure [options]
Options:
-h, --help
-d DBNAME, --database=DBNAME
Name of database for installed procedure
-f PROCPATH, --file=PROCPATH
Path of procedure file to install
-p OWNERPASSWORD, --password=OWNERPASSWORD
Password of procedure file owner
------------------------------------------------------------------------Usage: kill_host [options]
Options:
-h, --help
-s HOSTS, --hosts=HOSTS
comma-separated list of hosts on which the vertica
process is to be killed using a SIGKILL signal
Administrator's Guide
Using the Administration Tools
--compat21
hostnames
------------------------------------------------------------------------Usage: kill_node [options]
Options:
-h, --help
-s HOSTS, --hosts=HOSTS
comma-separated list of hosts on which the vertica
process is to be killed using a SIGKILL signal
--compat21
hostnames
------------------------------------------------------------------------Usage: license_audit [options]
Options:
-h, --help
show this help message and exit
-d DATABASE, --database=DATABASE
Name of the database to audit
-p PASSWORD, --password=PASSWORD
Password for database admin
-q, --quiet
Do not print status messages.
-f FILE, --file=FILE
Output results to FILE.
------------------------------------------------------------------------Usage: list_allnodes [options]
Options:
-h, --help
-d DB, --database=DB
Administrator's Guide
Using the Administration Tools
-h, --help
-n NODENAME, --node=NODENAME
Name of the node to be listed
------------------------------------------------------------------------Usage: list_packages [options]
Options:
-h, --help
-d DBNAME, --dbname=DBNAME
database name
-p PASSWORD, --password=PASSWORD
database admin password
-P PACKAGE, --package=PACKAGE
specify package or 'all' or 'default'
------------------------------------------------------------------------Usage: logrotateconfig [options]
Options:
-h, --help
-d DBNAME, --dbname=DBNAME
database name
-r ROTATION, --rotation=ROTATION
set how often the log is rotated.[
daily|weekly|monthly ]
-s MAXLOGSZ, --maxsize=MAXLOGSZ
set maximum log size before rotation is forced.
-k KEEP, --keep=KEEP
-d DB, --database=DB
-d DBNAME, --dbname=DBNAME
database name
-k KSAFETY, --ksafety=KSAFETY
specify the new k value to use
-p PASSWORD, --password=PASSWORD
Administrator's Guide
Using the Administration Tools
--script
later use.
------------------------------------------------------------------------Usage: restart_db [options]
Options:
-h, --help
-d DB, --database=DB
-e EPOCH, --epoch=EPOCH
Epoch at which the database is to be restarted. If
'last' is given as argument the db is restarted from
the last good epoch.
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-i, --noprompts
-s NODES, --hosts=NODES
comma-separated list of hosts to be restarted
-d DB, --database=DB
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-i, --noprompts
-F, --force
--compat21
hostnames
------------------------------------------------------------------------Usage: return_epoch [options]
Options:
-h, --help
-d DB, --database=DB
Name of database
-d DB, --database=DB
-p POLICY, --policy=POLICY
Restart policy: ('never', 'ksafe', 'always')
Administrator's Guide
Using the Administration Tools
-d DB, --database=DB
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-i, --noprompts
-F, --force
-d DB, --database=DB
-p DBPASSWORD, --password=DBPASSWORD
Database password in single quotes
-F, --force
connected.
-i, --noprompts
-s HOSTS, --hosts=HOSTS
comma-separated list of hosts on which the vertica
process is to be killed using a SIGTERM signal
--compat21
hostnames
------------------------------------------------------------------------Usage: stop_node [options]
Options:
-h, --help
-s HOSTS, --hosts=HOSTS
comma-separated list of hosts on which the vertica
Administrator's Guide
Using the Administration Tools
hostnames
------------------------------------------------------------------------Usage: uninstall_package [options]
Options:
-h, --help
-d DBNAME, --dbname=DBNAME
database name
-p PASSWORD, --password=PASSWORD
database admin password
-P PACKAGE, --package=PACKAGE
specify package or 'all' or 'default'
------------------------------------------------------------------------Usage: upgrade_license_key [options]
Options:
-h, --help
-d DB, --database=DB
-l LICENSE, --license=LICENSE
Database license
-i INSTALL, --install=INSTALL
argument '-i install' to Install license else without
'-i install' Upgrade license
-p PASSWORD, --password=PASSWORD
Database password[optional]
------------------------------------------------------------------------Usage: view_cluster [options]
Options:
-h, --help
-x, --xpand
-d DB, --database=DB
Administrator's Guide
Using Management Console
Connecting to MC
To connect to Management Console:
1. Open an HTML-5 compliant browser.
2. Enter the IP address or host name of the host on which you installed MC (or any cluster node if
you installed HP Vertica first), followed by the MC port you assigned when you configured MC
.
For example, enter one of:
https://00.00.00.00:5450/
or
https://hostname:5450/
3. When the MC logon dialog appears, enter your MC username and password and click Log in.
Administrator's Guide
Using Management Console
Overview, Activity and Node details pages could be limited. See About MC Privileges and
Roles for more information.
See Also
l
Administrator's Guide
Using Management Console
Note: Remove does not drop the database; it leaves it in the cluster, hidden from the UI. To
add the database back to the MC interface, import it using the IP address of any cluster
node. A Remove operation also stops metrics gathering on that database, but statistics
gathering automatically resumes after you re-import.
Drop the database after you ensure no users are connected. Drop is a permanent action that
drops the database from the cluster.
View Database to open the Overview page, a layout that provides a dashboard view into the
health of your database cluster (node state, storage, performance, CPU/memory, and query
concurrency). From this page you can drill down into more detailed database-specific
information by clicking data points in the graphs.
View Cluster to open the Manage page, which shows all nodes in the cluster, as well as each
node's state. You can also see a list of monitored databases on the selected cluster and its
state; for example, a green arrow indicates a database in an UP state. For node-specific
information, click any node to open the Node Details page.
For more information about what users can see and do on MC, see the following topics:
Administrator's Guide
Using Management Console
See Also
l
About MC Users
Administrator's Guide
Using Management Console
Database name and password. See Creating a Database Name and Password for rules.
Optionally click Advanced to open the advanced settings and change the port and catalog,
data, and temporary data paths. By default the MC application/web server port is 5450 and
paths are /home/dbadmin, or whatever you defined for the paths when you ran the Cluster
Creation Wizard or the install_vertica script. Do not use the default agent port 5444 as a
new setting for the MC port. See MC Settings > Configuration for port values.
5. Click Continue.
6. Select nodes to include in the database.
The Database Configuration window opens with the options you provided and a graphical
representation of the nodes appears on the page. By default, all nodes are selected to be part of
this database (denoted by a green check mark). You can optionally click each node and clear
Include host in new database to exclude that node from the database. Excluded nodes are
gray. If you change your mind, click the node and select the Include check box.
7. Click Create in the Database Configuration window to create the database on the nodes.
The creation process takes a few moments, after which the database starts and a Success
message appears on the interface.
8. Click OK to close the success message.
MC's Manage page opens and displays the database nodes. Nodes not included in the database
are colored gray, which means they are standby nodes you can include later. To add nodes to or
remove nodes from your HP Vertica cluster, which are not shown in standby mode, you must run
the install_vertica script.
Notes
l
If warnings occur during database creation, nodes will be marked on the UI with an Alert icon
and a message.
n
Warnings do not prevent the database from being created, but you should address warnings
after the database creation process completes by viewing the database Message Center
from the MC Home page.
Administrator's Guide
Using Management Console
Failure messages display on the database Manage page with a link to more detailed
information and a hint with an actionable task that you must complete before you can
continue. Problem nodes are colored red for quick identification.
To view more detailed information about a node in the cluster, double-click the node from the
Manage page, which opens the Node Details page.
To create MC users and grant them access to an MC-managed database, see About MC Users
and Creating an MC User.
See Also
l
Restarting MC
Administrator's Guide
Using Management Console
Administrator's Guide
Using Management Console
Enable Ganglia on your Hadoop cluster, to get the most information from your Hadoop
environment
2. In the Provision Access within Hadoop Environment window, enter a new Ambari
connection with your user name and password. If you already have an existing connection, a
window opens that allows you to select your existing connection from the drop-down menu.
3. In the next window, select the Hadoop cluster where the HP Vertica database you want to
monitor resides. MC will discover Hadoop clusters that are currently monitored by the Ambari
Administrator's Guide
Using Management Console
7. Confirm that the import is successful. If it is, a success message appears. Click Done to go to
the Existing Infrastructure page.
To import an additional HP Vertica cluster within a Hadoop environment, click Import Cluster or
database using IP address discovery under Provisioning. Management Console will
automatically associate the cluster with the existing Hadoop environment.
Administrator's Guide
Using Management Console
Click to select the Hadoop environment, and then click View Vertica Databases.
The resulting screen displays information about the HP Vertica databases that reside in a Hadoop
environment:
Administrator's Guide
Using Management Console
You can monitor information such as resource usage, Hadoop services, and database and
connection status.
See Also:
HPVertica for SQLon Hadoop
Managing MC Settings
The MC Settings page allows you to configure properties specific to Management Console. You
can:
Administrator's Guide
Using Management Console
Enable checks and set alert thresholds for spread retransmit rate. This setting is disabled by
default. The recommended alert threshold for spread retransmit rate is 10%.
Set alert thresholds for free MCdisk space checks. The recommended alert threshold is 500
MB.
Create new MC users and map them to an MC-managed database using user credentials on the
HP Vertica server.
Administrator's Guide
Using Management Console
See Also
l
Administrator's Guide
Using Management Console
7. Repeat (as root) Step 6 on each cluster node where you copied the config.py file.
OR
https://hostname:5555/
Administrator's Guide
Using Management Console
Backing Up MC
Before you upgrade MC, HP recommends that you back up your MC metadata (configuration and
user settings) on a storage location external to the server on which you installed MC.
1. On the target server (where you want to store MC metadata), log on as root or a user with sudo
privileges.
2. Create a backup directory; for example:
# mkdir /backups/mc/mc-backup-20130425
Administrator's Guide
Using Management Console
Resource Usages in Pool: Shows the historically averaged acquired memory usage by each
pool across all nodes. The graph uses two y-axes, one that shows memory size, and a second
that shows the total number of running queries. Data is collected every hour. Hover over a data
point for a summary of the memory usage at that specific point.
Memory Usage in Node: Shows the historically acquired memory usages by all pools across
all nodes. Data is collected every hour. Hover over a data point for a summary of the memory
usage at that specific point
Average Query Execution and Query Time in Pool: Shows the averaged query queue time
plotted against the query execution time by each pool across all nodes. Data is collected every
minute. Hover over data to get the average query execution and queue time in the specified pool.
Click a data point to show detailed individual query information.
Resource Rejections in Pool: Shows the historical total number of resource requests that
were rejected in each pool across all nodes. Data is collected every hour. Click a data point to
show rejection details and reasons in a pop-up window.
Administrator's Guide
Using Management Console
Click in a parameter field to display more information about that parameter. Once you make a
change, click Apply. A success message appears if the change was made.
See Also
l
Administrator's Guide
Using Management Console
View Management Console logs, which you can sort by column headings, such as type,
component, or message).
Search within messages for key words or phrases and search for log entries within a specific
time frame.
Caution: Reset removes all data (monitoring and configuration information) from storage
and forces you to reconfigure MC as if it were the first time.
Restart the Management Console process. When the process completes, you are directed back
to the login page.
Administrator's Guide
Using Management Console
See Also
l
Administrator's Guide
Using Management Console
Restarting MC
You might need to restart the MC web/application server for a number of reasons, such as after you
change port assignments, use the MC interface to import a new SSL certificate, or if the MC
interface or HP Vertica-related tasks become unresponsive.
Restarting MC requires ADMIN Role (mc) or SUPER Role (mc) privileges.
stop
start
Administrator's Guide
Using Management Console
restart
Restarts the MC application/web server. This process will report that the stop
didn't work if MC is not already running.
Starting over
If you need to return MC to its original state (a "factory reset"), see Resetting MC to Pre-Configured
State.
When you first configure MC, during the configuration process you create an MC super user (a
Linux account). Issuing a Factory Reset on the MC does not create a new MC super user, nor
does it delete the existing MC super user. When initializing after a Factory Reset, you must
logon using the original MC super user account.
Note that, once MC is configured, you can add users that are specific to MC. Users created
through the MC interface are MC specific. When you subsequently change a password through
the MC, you only change the password for the specific MC user. Passwords external to MC
(i.e., system Linux users and HP Vertica database passwords) remain unchanged.
For information on MC users, refer to the sections, Creating an MC User and MC configuration
privileges.
Administrator's Guide
Using Management Console
replay attacks, the timestamp is valid for several seconds only, after which it expires.
To avoid being blocked out of MC, synchronize time on the hosts in your HP Vertica cluster, and on
the MC host if it resides on a dedicated server. To recover from loss or lack of synchronization,
resync system time and the Network Time Protocol. See Set Up Time Synchronization in the
Installation Guide. If you want to generate your own certificates and keys for MC, see Generating
Certificates and Keys for MC.
Administrator's Guide
Using Management Console
Administrator's Guide
Operating the Database
Administrator's Guide
Operating the Database
Caution: HP strongly recommends that you start only one database at a time. If you start
more than one database at any time, the results are unpredictable. Users could encounter
resource conflicts or perform operations in the wrong database.
Administrator's Guide
Operating the Database
Note: You cannot stop a running database if any users are connected or Database Designer is
building or deploying a database design.
Tip: To check for open sessions, query the V_MONITOR.SESSIONS table. The client_
label column returns a value of MC for users who are connected to MC.
Administrator's Guide
Operating the Database
3. Close any remaining user sessions. (Use the CLOSE_SESSION and CLOSE_ALL_
SESSIONS functions.)
4. Open the Administration Tools. See Using the Administration Tools for information about
accessing the Administration Tools.
5. On the Main Menu, select Stop Database, and then click OK.
6. Select the database you want to stop, and click OK.
7. Enter the password if asked, and click OK.
8. When prompted that the database has been successfully stopped, click OK.
As long as the license is valid, the -p argument is not required to stop the database and is silently
ignored, even if you introduce a typo or press the enter key prematurely. This is by design, as the
database can only be stopped by the user who (as part of the verticadba UNIX user group) initially
created the database or who has root or su privileges.
If the license were to become invalid, HP Vertica would use the -p password argument to attempt
to upgrade the license with the license file stored in /opt/vertica/config/share/license.key.
Administrator's Guide
Operating the Database
Run a per-block cyclic redundancy check (CRC) on data storage to verify data integrity.
If the database is down, invoke the Index tool from the Linux command line. If the database is up,
invoke it as an SQL statement from vsql:
Operation
Database Down
Database Up
Run CRC
/opt/vertica/bin/vertica
-Dcatalog-path-v
Checksortor
/opt/vertica/bin/vertica
-Dcatalog-path-I
der
If you run the Index tool in vsql as an SQL statement, you can specify that it analyze all cluster
nodes by setting the optional Boolean parameter to true (1). If this parameter is omitted, the Index
tool runs only on the current node.
If invoked from the command line, the Index tool runs only on the current node. However, the Index
tool can run on multiple nodes simultaneously. Invoke the Index tool binary from the
/opt/vertica/bin directory.
Viewing Results
The Index tool writes summary information about its operation to standard output; detailed
information on results is logged in one of two locations, depending on the environment where you
invoke the tool:
l
Invoked from the command-line: Results written to indextool.log file in the database
catalog directory.
Privileges
Restricted to superusers.
Administrator's Guide
Operating the Database
The Index tool writes summary information about its operation to standard output, and logs
detailed information in indextool.log in the database catalog directory.
l
The Index tool writes summary information about its operation to standard output, and logs
detailed information in vertica.log on the current node.
Administrator's Guide
Operating the Database
The Event Manager is also notified of CRC errors, so you can use an SNMP trap to capture CRC
errors:
"CRC mismatch detected on file <file_path>. File may be corrupted. Please check hardware
and drivers."
If you run a query from vsql, ODBC, or JDBC, the query returns a FileColumnReader ERROR. This
message indicates that a specific block's CRC does not match a given record as follows:
hint: Data file may be corrupt. Ensure that all hardware (disk and memory) is working
properly. Possible solutions are to delete the file <pathname> while the node is down,
and then allow the node to recover, or truncate the table data.code: ERRCODE_DATA_
CORRUPTED
The Index tool writes summary information about its operation to standard output, and logs
detailed information in indextool.log in the database catalog directory.
l
Administrator's Guide
Operating the Database
Invoke the Index tool from vsql as an SQL statement with the argument checkcrc. To run the
index tool on all nodes, also set the tool's optional Boolean parameter to true. If this parameter
is omitted, the Index tool runs only on the current node.
For example, the following SQL statement runs a CRCon all cluster nodes:
select run_index_tool ('checksort', 'true');
The Index tool writes summary information about its operation to standard output, and logs
detailed information in vertica.log on the current node.
Reviewing Errors
1. Open the indextool.log file. For example:
$cd VMart/v_check_node0001_catalog
2. Look for error messages that include an OID number and the string Sort Order Violation.
For example:
<INFO> ...on oid 45035996273723545: Sort Order Violation:
3. Find detailed information about the sort order violation string by running grep on
indextool.log. For example, the following command returns the line before each string (-B1),
and the four lines that follow (-A4):
[15:07:55][vertica-s1]: grep -B1 -A4 'Sort Order Violation:' /my_
host/databases/check/v_check_node0001_catalog/indextool.log
2012-06-14 14:07:13.686 unknown:0x7fe1da7a1950 [EE] <INFO> An error occurred when
running index tool thread on oid 45035996273723537:
Sort Order Violation:
Row Position: 624
Column Index: 0
Last Row: 2576000
This Row: 2575000
-2012-06-14 14:07:13.687 unknown:0x7fe1dafa2950 [EE] <INFO> An error occurred when
running index tool thread on oid 45035996273723545:
Sort Order Violation:
Row Position: 3
Column Index: 0
Last Row: 4
This Row: 2
--
Administrator's Guide
Operating the Database
4. Find the projection where a sort order violation occurred by querying the storage_containers
system table. Use a storage_oid equal to the OID value listed in indextool.log. For
example:
=> select * from storage_containers where storage_oid = 45035996273723545;
Administrator's Guide
Operating the Database
Administrator's Guide
Managing Tables
Managing Tables
You can create two types of tables in HP Vertica, columnar and flexible. Additionally, you can
create either type as persistent or temporary. You can also create views to capture a specific set of
table columns that you query frequently.
This section describes how to:
l
Alter tables
Administrator's Guide
Managing Tables
Create a new table, StateBird, with the /*+direct*/ clause in the statement, placing the clause
directly before the query (select State, Bird from states):
VMart=> create table StateBird as /*+direct*/ select State, Bird from states;
CREATE TABLE
VMart=> select * from StateBird;
State |
Bird
-------+---------MA
| Robin
NH
| Thrush
NY
| Cardinal
(3 rows)
The following example creates a temporary table using the /*+direct*/ clause, along with the ON
COMMIT PRESERVE ROWS directive:
VVMart=> create temp table StateTax ON COMMIT PRESERVE ROWS as /*+direct*/ select State,
Tax from states;
CREATE TABLE
VMart=> select * from StateTax;
State | Tax
-------+----MA
| 5.7
NH
|
0
NY
| 7.2
(3 rows)
If you create a table without providing the projection-related clauses, HP Vertica automatically
creates a superprojection for the table when you use an INSERT INTO or COPY statement to
Administrator's Guide
Managing Tables
load data into the table for the first time. The projection is created in the same schema as the
table. Once HP Vertica has created the projection, it loads the data.
l
If you use CREATE TABLE AS SELECT to create a table from the results of a query, the table
is created first and a projection is created immediately after, using some of the properties of the
underlying SELECT query.
(Advanced users only) If you use any of the following parameters, the default projection is
created immediately upon table creation using the specified properties:
n
ORDER BY table-column
hash-segmentation-clause
KSAFE
Note: Before you define a superprojection in the above manner, read Creating Custom
Designs in the Administrator's Guide.
It is a superprojection.
If created as a result of a CREATE TABLE AS SELECT statement, uses the encoding specified
in the query table.
The number of table columns used in the segmentation expression can be configured, using the
MaxAutoSegColumns configuration parameter. See General Parameters in the Administrator's
Guide. Columns are segmented in this order:
Administrator's Guide
Managing Tables
Auto-projections are defined by the table properties and creation methods, as follows:
If table...
Segmentation is...
Same as input
stream (COPY or
stream, if sorted.
INSERT INTO)
the table
Same as input
TABLE AS SELECT
stream, if sorted.
segmented
If not sorted,
sorted using
unsegmented or unknown
query
following rules.
Has FK and PK
FK first, then PK
PK columns
constraints
columns
FK first, then
(no PK)
remaining columns
PK columns
PK columns
On all columns
(no FK)
Has no FK or PK
constraints
Default automatic projections and segmentation get your database up and running quickly. HP
recommends that you start with these projections and then use the Database Designer to optimize
your database further. The Database Designer creates projections that optimize your database
based on the characteristics of the data and, optionally, the queries you use.
Administrator's Guide
Managing Tables
See Also
l
Projection Concepts
CREATE TABLE
Administrator's Guide
Managing Tables
Simple Example
This example shows how to use the statement for a table that already exists, and suggests a
naming convention that describes the contents of the new table:
Create a new schema in which to create an intermediate table with projections. This is the table into
which you will move partitions. Then, create a table identical to the source table from which to move
partitions:
VMART=> create schema partn_backup;
CREATE SCHEMA
VMART=> create table partn_backup.trades_200801 like prod.trades including projections;
CREATE TABLE
Once the schema and table exist, you can move one or more of the existing table partitions to the
new intermediate table.
Administrator's Guide
Managing Tables
tax
See Also
l
Moving Partitions
CREATE TABLE
Administrator's Guide
Managing Tables
Administrator's Guide
Managing Tables
as described in the section Automatic Projection Creation in CREATE TABLE. Also, while you
can add projections for tables created with the ON COMMIT DELETE ROWS option, be aware
that you could save the projection but still lose all the data.
Create a new table, StateBird, with the /*+direct*/ clause in the statement, placing the clause
directly before the query (select State, Bird from states):
VMart=> create table StateBird as /*+direct*/ select State, Bird from states;
CREATE TABLE
VMart=> select * from StateBird;
State |
Bird
-------+---------MA
| Robin
NH
| Thrush
NY
| Cardinal
(3 rows)
The following example creates a temporary table using the /*+direct*/ clause, along with the ON
COMMIT PRESERVE ROWS directive:
VVMart=> create temp table StateTax ON COMMIT PRESERVE ROWS as /*+direct*/ select State,
Tax from states;
CREATE TABLE
VMart=> select * from StateTax;
State | Tax
-------+----MA
| 5.7
NH
|
0
NY
| 7.2
(3 rows)
Administrator's Guide
Managing Tables
It is a superprojection.
Temp tables are not recoverable, so the superprojection is not K-Safe (K-SAFE=0), and you
cannot make it so.
Auto-projections are defined by the table properties and creation methods, as follows:
If table...
Segmentation is...
Same as input
stream (COPY or
stream, if sorted.
INSERT INTO)
the table
Same as input
TABLE AS SELECT
stream, if sorted.
segmented
If not sorted,
sorted using
unsegmented or unknown
query
following rules.
Has FK and PK
FK first, then PK
PK columns
constraints
columns
FK first, then
(no PK)
remaining columns
PK columns
PK columns
On all columns
(no FK)
Has no FK or PK
constraints
Administrator's Guide
Managing Tables
Advanced users can modify the default projection created through the CREATE TEMPORARY TABLE
statement by defining one or more of the following parameters:
l
ORDER BY table-column
hash-segmentation-clause
NO PROJECTION
Note: Before you define the superprojection in this manner, read Creating Custom Designs in
the Administrator's Guide.
NUMERIC, y
NUMERIC )
To preserve temporary table data until the end of the session, use the ON COMMIT PRESERVE
ROWS clause:
=> CREATE GLOBAL TEMP TABLE temp_table2 (x
ON COMMIT PRESERVE ROWS;
NUMERIC, y
NUMERIC )
NUMERIC, y
Administrator's Guide
Managing Tables
The following example specifies that the superprojection created for the temp table use the sort
order specified by the ORDER BY clause, rather than the order of columns in the column list.
=> CREATE GLOBAL TEMP TABLE temp_table1 (
x NUMERIC,
y NUMERIC ENCODING RLE,
b VARCHAR(8),
z VARCHAR(8) )
ORDER BY z, x;
See Also
l
Projection Concepts
CREATE TABLE
TRUNCATE TABLE
DELETE
ANALYZE_STATISTICS
Administrator's Guide
Managing Tables
so on.
Note: Permission requirements for external tables differ from other tables. To gain full access
(including SELECT) to an external table that a user has privileges to create, the database
superuser must also grant READ access to the USER-accessible storage location, see
GRANT (Storage Location).
Examples
Examples of external table definitions:
CREATE EXTERNAL TABLE ext1 (x integer) AS COPY FROM '/tmp/ext1.dat' DELIMITER ',';
CREATE EXTERNAL TABLE ext1 (x integer) AS COPY FROM '/tmp/ext1.dat.bz2' BZIP DELIMITER
',';
CREATE EXTERNAL TABLE ext1 (x integer, y integer) AS COPY (x as '5', y) FROM
'/tmp/ext1.dat.bz2' BZIP DELIMITER ',';
See Also
l
COPY
Administrator's Guide
Managing Tables
Administrator's Guide
Managing Tables
2. Use a query such as the following to list the external table definitions (table_definition):
select table_name, table_definition from v_catalog.tables;
table_name |
table_definition
------------+---------------------------------------------------------------------t1
| COPY
FROM 'TMPDIR/external_table.dat' DELIMITER ','
t1_copy
| COPY
FROM 'TMPDIR/external_table.dat' DELIMITER ','
t2
| COPY FROM 'TMPDIR/external_table2.dat' DELIMITER ','
(3 rows)
Administrator's Guide
Managing Tables
Supported
Unsupported
Administrator's Guide
Managing Tables
8
10
11
12
(11 rows)
7. Join the external table (ext) with the table created in HP Vertica, called second:
=> SELECT * FROMext JOIN second ON x=y;
x | y
---+--1 | 1
1 | 1
3 | 3
4 | 4
5 | 5
(5 rows)
Administrator's Guide
Managing Tables
For more details about creating an external table, see CREATE EXTERNAL TABLE statement.
The data you specify in the FROM clause of a CREATE EXTERNAL TABLE AS COPY statement
can be stored in one or more files or directories, and on one or more nodes. After successfully
creating an external table, HP Vertica stores the table name and its COPY definition. Each time a
SELECT query references the external table, HP Vertica parses the COPY statement definition
again to access the data. Here is a sample select statement:
=>SELECT * FROM tbl WHERE i > 10;
Administrator's Guide
Managing Tables
In this case, the external table COPY statement will include a wildcard definition such as the
following:
CREATE TABLE archive_data (...) AS COPY FROM 'nfs_name/2012/monthly_archived_data/*'
Whenever an HP Vertica query references the external table months, and HP Vertica parses the
COPY statement, all stored data tables in the top-level monthly_archived_data directory are
made accessible to the query.
Rename a table
Administrator's Guide
Managing Tables
ADD COLUMN
RENAME COLUMN
SET SCHEMA
PARTITION BY
REORGANIZE
REMOVE PARTITIONING
RENAME [TO ]
OWNER TO
Note: You can use the ADD constraints and DROP constraints clauses together.
Administrator's Guide
Managing Tables
Inserts the default value for existing rows. For example, if the default expression is CURRENT_
TIMESTAMP, all rows have the current timestamp.
Automatically adds the new column with a unique projection column name to the
superprojection of the table.
You can add a column to a table using the ALTER TABLE ADD COLUMN statement with the
CASCADE keyword. When you use CASCADE, HP Vertica also adds the new table column to all
pre-join projections that are created using that table.
If you use the CASCADE keyword and specify a nonconstant default column value, HP Vertica
does not add the column to the pre-join projections.
For detailed information about how to add columns to tables, see ALTER TABLE.
Note: Adding a column to a table does not affect the K-safety of the physical schema design,
and you can add columns when nodes are down.
Administrator's Guide
Managing Tables
As the default expression for an ALTER TABLE ALTER COLUMN statement on an external table:
ALTER TABLE mytable ADD COLUMN a2 FLOAT DEFAULT RANDOM();
ROLLBACK 5241: Unsupported access to external table
ALTER TABLE mytable ALTER COLUMN x SET DEFAULT RANDOM();
ROLLBACK 5241: Unsupported access to external table
You can specify a volatile function as a column default expression using the ALTER TABLE ALTER
COLUMN statement:
ALTER TABLE t ALTER COLUMN a2 SET DEFAULT my_sequence.nextval;
Administrator's Guide
Managing Tables
After a DROP COLUMN operation completes, data backed up from the current epoch onward
will recover without the column. Data recovered from a backup prior to the current epoch will readd the table column. Because drop operations physically purge object storage and catalog
definitions (table history) from the table, AT EPOCH (historical) queries return nothing for the
dropped column.
If you change a column's data type from CHAR(8) to CHAR(16) in epoch 10 and you restore the
database from epoch 5, the column will be CHAR(8) again.
Administrator's Guide
Managing Tables
You cannot specify a default expression that derives data from another derived column. This means
that if you already have a column with a default derived value expression, you cannot add another
column whose default references the existing column.
Note: You can add a column when nodes are down.
3. Use ALTER TABLE to add a fourth column that extracts the month from the timestamp value in
column a:
=> ALTER TABLE t ADD COLUMN d INTEGER DEFAULT EXTRACT(MONTH FROM a);
ALTER TABLE
4. Query table t:
=> select * from t;
Administrator's Guide
Managing Tables
a
| b | c | d
---------------------+---+----+--2012-05-14 10:39:25 | 2 | MA | 5
(1 row)
Administrator's Guide
Managing Tables
4. Use ALTER TABLE to add a column to t1 with the default column value derived from the
UDSF, add2ints:
alter table t1 add column z int default add2ints(x,y);
ALTER TABLE
Binary typesexpansion and contraction but cannot convert between BINARY and
VARBINARY types.
Exact numeric typesINTEGER, INT, BIGINT, TINYINT, INT8, SMALLINT, and all
NUMERIC values of scale <=18 and precision 0 are interchangeable. For NUMERIC data
types, you cannot alter precision, but you can change the scale in the ranges (0-18), (19-37), and
so on.
HP Vertica does not allow data type conversion on types that require storage reorganization:
Administrator's Guide
Managing Tables
You can expand (and shrink) columns within the same class of data type, which is useful if you
want to store longer strings in a column. HP Vertica validates the data before it performs the
conversion.
For example, if you try to convert a column from varchar(25) to varchar(10) and that column holds a
string with 20 characters, the conversion will fail. HP Vertica allow the conversion as long as that
column does not have a string larger than 10 characters.
Examples
The following example expands an existing CHAR column from 5 to 10:
=> CREATE TABLE t (x CHAR, y CHAR(5));
=> ALTER TABLE t ALTER COLUMN y SET DATA TYPE CHAR(10);
=> DROP TABLE t;
This example illustrates the behavior of a changed column's type. First you set column y's type to
VARCHAR(5) and then insert strings with characters that equal 5 and exceed 5:
=> CREATE TABLE t (x VARCHAR, y VARCHAR);
=> ALTER TABLE t ALTER COLUMN y SET DATA TYPE VARCHAR(5);
=> INSERT INTO t VALUES ('1232344455','hello');
OUTPUT
-------1
(1 row)
=> INSERT INTO t VALUES ('1232344455','hello1');
ERROR 4797: String of 6 octets is too long for type Varchar(5)
=> DROP TABLE t;
You can also contract the data type's size, as long as that altered column contains no strings
greater than 5:
=> CREATE TABLE t (x CHAR, y CHAR(10));
=> ALTER TABLE t ALTER COLUMN y SET DATA TYPE CHAR(5);
=> DROP TABLE t;
Administrator's Guide
Managing Tables
You cannot convert types between binary and varbinary. For example, the table definition below
contains two binary columns, so when you try to convert column y to a varbinary type, HP Vertica
returns a ROLLBACK message:
=> CREATE TABLE t (x BINARY, y BINARY);
=> ALTER TABLE t ALTER COLUMN y SET DATA TYPE VARBINARY;--N
ROLLBACK 2377: Cannot convert column "y" from "binary(1)" to type "varbinary(80)
=> DROP TABLE t;
Administrator's Guide
Managing Tables
2)::NUMERIC;ALTER TABLE
5. Query the sales table, and you'll see the new temp_price column with its derived NUMERIC
values:
=> SELECT * FROM SALES;
id | price |
temp_price
----+---------+--------------------1 | $50.00 | 50.000000000000000
2 | $100.00 | 100.000000000000000
(2 rows)
8. Manage epochs:
SELECT manage_epoch();
manage_epoch
-------------------------------Current Epoch=83, AHM Epoch=82
10. Rename the new (temporary) temp_price column back to its original name, price:
=> ALTER TABLE sales RENAME COLUMN temp_price to price;
Administrator's Guide
Managing Tables
price
Examples
ALTER TABLE T1 ALTER COLUMN x SET NOT NULL;
ALTER TABLE T1 ALTER COLUMN x DROP NOT NULL;
Note: Using the [SET | DROP] NOT NULL clause does not validate whether the column data
conforms to the NOT NULL constraint. Use ANALYZE_CONSTRAINTS to check for
constraint violations in a table.
Administrator's Guide
Managing Tables
See Also
l
About Constraints
Because drop operations physically purge object storage and catalog definitions (table history) from
the table, AT EPOCH (historical) queries return nothing for the dropped column.
The altered table has the same object ID.
Note: Drop column operations can be fast because these catalog-level changes do not require
data reorganization, letting you quickly reclaim disk storage.
Restrictions
l
At the table level, you cannot drop or alter a primary key column or a column participating in the
table's partitioning clause.
At the projection level, you cannot drop the first column in a projection's sort order or columns
that participate in in the segmentation expression of a projection.
Administrator's Guide
Managing Tables
Column to
drop ...
Has a constraint
HP Vertica will drop the column with CASCADE specified when a FOREIGN
of any kind on it
Participates in
If you drop a column using the CASCADE keyword, HP Vertica truncates the
the projection's
projection's sort order up to and including the projection that has been dropped
sort order
without impact on physical storage for other columns and then drops the
specified column. For example if a projection's columns are in sort order
(a,b,c), dropping column b causes the projection's sort order to be just (a),
omitting column (c).
Participates in
ONE of the
maintains the superprojection that contains the data, and drops the
following:
Is a pre-join
projection
Assume, for example, that you have a table with multiple projections and
where one projection has the column you are trying to drop is part of the
Participates
in the
statement tries to implicitly drop the projection that has this column in the
projection's
Examples
The following series of commands successfully drops a BYTEA data type column:
=> CREATE TABLE t (x BYTEA(65000), y BYTEA, z BYTEA(1));
CREATE TABLE
=> ALTER TABLE t DROP COLUMN y;
ALTER TABLE
=> SELECT y FROM t;
ERROR 2624: Column "y" does not exist
=> ALTER TABLE t DROP COLUMN x RESTRICT;
ALTER TABLE
Administrator's Guide
Managing Tables
The following series of commands tries to drop a FLOAT(8) column and fails because there are not
enough projections to maintain K-safety.
=> CREATE TABLE t (x FLOAT(8),y FLOAT(08));
CREATE TABLE
=> ALTER TABLE t DROP COLUMN y RESTRICT;
ALTER TABLE
=> SELECT y FROM t;
ERROR 2624: Column "y" does not exist
=> ALTER TABLE t DROP x CASCADE;
ROLLBACK 2409: Cannot drop any more columns in t
=> DROP TABLE t CASCADE;
Note: HP Vertica lets you move system tables to system schemas. Moving system tables
could be necessary to support designs created through the Database Designer.
Administrator's Guide
Managing Tables
In order to alter table ownership, you must be either the table owner or a superuser.
A change in table ownership transfers just the owner and not privileges; grants made by the original
owner are dropped and all existing privileges on the table are revoked from the previous owner.
However, altering the table owner transfers ownership of dependent sequence objects (associated
IDENTITY/AUTO-INCREMENT sequences) but does not transfer ownership of other referenced
sequences. See ALTER SEQUENCE for details on transferring sequence ownership.
Notes
l
Table privileges are separate from schema privileges; therefore, a table privilege change or table
owner change does not result in any schema privilege change.
Because projections define the physical representation of the table, HP Vertica does not require
separate projection owners. The ability to create or drop projections is based on the table
privileges on which the projection is anchored.
During the alter operation HP Vertica updates projections anchored on the table owned by the
old owner to reflect the new owner. For pre-join projection operations, HP Vertica checks for
privileges on the referenced table.
Example
In this example, user Bob connects to the database, looks up the tables, and transfers ownership of
table t33 from himself to to user Alice.
=> \c - BobYou are now connected as user "Bob".
=> \d
Schema | Name | Kind | Owner | Comment
--------+--------+-------+---------+--------public | applog | table | dbadmin |
public | t33
| table | Bob
|
(2 rows)
Administrator's Guide
Managing Tables
Notice that when Bob looks up database tables again, he no longer sees table t33.
=> \d
List of tables
List of tables
Schema | Name | Kind | Owner | Comment
--------+--------+-------+---------+--------public | applog | table | dbadmin |
(1 row)
When user Alice connects to the database and looks up tables, she sees she is the owner of table
t33.
=> \c - AliceYou are now connected as user "Alice".
=> \d
List of tables
Schema | Name | Kind | Owner | Comment
--------+------+-------+-------+--------public | t33 | table | Alice |
(2 rows)
Either Alice or a superuser can transfer table ownership back to Bob. In the following case a
superuser performs the transfer.
=> \c - dbadminYou are now connected as user "dbadmin".
=> ALTER TABLE t33 OWNER TO Bob;
ALTER TABLE
=> \d
List of tables
Schema |
Name
| Kind | Owner | Comment
--------+----------+-------+---------+--------public | applog
| table | dbadmin |
public | comments | table | dbadmin |
public | t33
| table | Bob
|
s1
| t1
| table | User1
|
(4 rows)
You can also query the V_CATALOG.TABLES system table to view table and owner information.
Note that a change in ownership does not change the table ID.
In the below series of commands, the superuser changes table ownership back to Alice and queries
the TABLES system table.
=> ALTER TABLE t33 OWNER TO Alice;
ALTER TABLE
=> SELECT table_schema_id, table_schema, table_id, table_name, owner_id, owner_name FROM
tables; table_schema_id | table_schema |
table_id
| table_name |
owner_id
Administrator's Guide
Managing Tables
| owner_name
-------------------+--------------+-------------------+------------+-------------------+----------45035996273704968 | public
| 45035996273713634 | applog
| 45035996273704962 |
dbadmin
45035996273704968 | public
| 45035996273724496 | comments
| 45035996273704962 |
dbadmin
45035996273730528 | s1
| 45035996273730548 | t1
| 45035996273730516 |
User1
45035996273704968 | public
| 45035996273795846 | t33
| 45035996273724576 |
Alice
(5 rows)
Now the superuser changes table ownership back to Bob and queries the TABLES table again.
Nothing changes but the owner_name row, from Alice to Bob.
=> ALTER TABLE t33 OWNER TO Bob;
ALTER TABLE
=> SELECT table_schema_id, table_schema, table_id, table_name, owner_id, owner_name FROM
tables;
table_schema_id | table_schema |
table_id
| table_name |
owner_id
|
owner_name-------------------+--------------+-------------------+------------+------------------+-----------45035996273704968 | public
| 45035996273713634 | applog
| 45035996273704962 |
dbadmin
45035996273704968 | public
| 45035996273724496 | comments
| 45035996273704962 |
dbadmin
45035996273730528 | s1
| 45035996273730548 | t1
| 45035996273730516 |
User1
45035996273704968 | public
| 45035996273793876 | foo
| 45035996273724576 |
Alice
45035996273704968 | public
| 45035996273795846 | t33
| 45035996273714428 |
Bob
(5 rows)
Administrator's Guide
Managing Tables
See Also
l
This operation immediately reassigns the sequence from the current owner to the specified new
owner.
Only the sequence owner or a superuser can change ownership, and reassignment does not
transfer grants from the original owner to the new owner; grants made by the original owner are
dropped.
Note: Renaming a table owner transfers ownership of dependent sequence objects
(associated IDENTITY/AUTO-INCREMENT sequences) but does not transfer ownership of other
referenced sequences. See Changing a Table Owner.
Administrator's Guide
Managing Tables
Example
The following example reassigns sequence ownership from the current owner to user Bob:
=> ALTER SEQUENCE sequential OWNER TO Bob;
Renaming Tables
The ALTER TABLE RENAME TO statement lets you rename one or more tables. The new table names
must not exist already.
Renaming tables does not affect existing pre-join projections because pre-join projections refer to
tables by their unique numeric object IDs (OIDs). Renaming tables also does not change the table
OID.
To rename two or more tables:
1. List the tables to rename with a comma-delimited list, specifying a schema-name after part of
the table specification only before the RENAME TO clause:
=> ALTER TABLE S1.T1, S1.T2 RENAME TO U1, U2;
The statement renames the listed tables to their new table names from left to right, matching
them sequentially, in a one-to-one correspondence.
The RENAME TO parameter is applied atomically so that all tables are renamed, or none of the
tables is renamed. For example, if the number of tables to rename does not match the number
of new names, none of the tables is renamed.
2. Do not specify a schema-name as part of the table specification after the RENAME TO clause,
since the statement applies to only one schema. The following example generates a syntax
error:
=> ALTER TABLE S1.T1, S1.T2 RENAME TO S1.U1, S1.U2;
Note: Renaming a table referenced by a view causes the view to fail, unless you create
another table with the previous name to replace the renamed table.
Administrator's Guide
Managing Tables
2. Use the RENAME TO clause to swap the tables: T1 to temps, T2 to T1, and temps to T2:
RENAME TO temps, T1, T2;
Administrator's Guide
Managing Tables
Auto-increment column value: a sequence available only for numeric column types. Autoincrement sequences automatically assign the next incremental sequence value for that column
when a new row is added to the table.
Auto-increment and Identity sequences are defined through column constraints in the CREATE
TABLE statement and are incremented each time a row is added to the table. Both of these object
types are table-dependent and do not persist independently. The identity value is never rolled back
even if the transaction that tries to insert a value into the table is not committed. The LAST_
INSERT_ID function returns the last value generated for an auto-increment or identity column.
Each type of sequence object has a set of properties and controls. A named sequence has the most
controls, and an Auto-increment sequence the least. The following table lists the major differences
between the three sequence objects:
Behavior
Administrator's Guide
Managing Tables
Behavior
Requires name
Use in expressions
Change parameters
Always starts at 1
While sequence object values are guaranteed to be unique, they are not guaranteed to be
contiguous. Since sequences are not necessarily contiguous, you may interpret the returned values
as missing. For example, two nodes can increment a sequence at different rates. The node with a
heavier processing load increments the sequence, but the values are not contiguous with those
being incremented on the other node.
Administrator's Guide
Managing Tables
NEXTVAL Advances the sequence and returns the next sequence value. This value is
incremented for ascending sequences and decremented for descending sequences. The first
time you call NEXTVAL after creating a sequence, the function sets up the cache in which to
store the sequence values, and returns either the default sequence value, or the start number
you specified with CREATE SEQUENCE.
CURRVAL Returns the LAST value across all nodes returned by a previous invocation of
NEXTVAL in the same session. If there were no calls to NEXTVAL after creating a sequence,
the CURRVAL function returns an error:
dbt=> create sequence seq2;
CREATE SEQUENCE
dbt=> select currval('seq2');
ERROR 4700: Sequence seq2 has not been accessed in the session
You can use the NEXTVAL and CURRVAL functions in INSERT and COPY expressions.
To...
CREATE
SEQUENCE
ALTER
SEQUENCE
DROP
SEQUENCE
GRANT
SEQUENCE
Privileges.
Administrator's Guide
Managing Tables
Creating Sequences
Create a sequence using the CREATE SEQUENCE statement. All of the parameters (besides a
sequence name) are optional.
The following example creates an ascending sequence called my_seq, starting at 100:
dbt=> create sequence my_seq START 100;
CREATE SEQUENCE
After creating a sequence, you must call the NEXTVAL function at least once in a session to create
a cache for the sequence and its initial value. Subsequently, use NEXTVAL to increment the
sequence. Use the CURRVAL function to get the current value.
The following NEXTVAL function instantiates the newly-created my_seq sequence and sets its first
number:
=> SELECT NEXTVAL('my_seq');
nextval
--------100
(1 row)
The following command returns the current value of this sequence. Since no other operations have
been performed on the newly-created sequence, the function returns the expected value of 100:
=> SELECT CURRVAL('my_seq');
currval
--------100
(1 row)
Calling the CURRVAL again function returns only the current value:
Administrator's Guide
Managing Tables
The following example shows how to use the my_seq sequence in an INSERT statement.
=> CREATE TABLE customer (
lname VARCHAR(25),
fname VARCHAR(25),
membership_card INTEGER,
id INTEGER
);
=> INSERT INTO customer VALUES ('Hawkins' ,'John', 072753, NEXTVAL('my_seq'));
Now query the table you just created to confirm that the ID column has been incremented to 102:
=> SELECT * FROM customer;
lname | fname | membership_card | id
---------+-------+-----------------+----Hawkins | John |
72753 | 102
(1 row)
The following example shows how to use a sequence as the default value for an INSERT
command:
=> CREATE TABLE customer2(
id INTEGER DEFAULT NEXTVAL('my_seq'),
lname VARCHAR(25),
fname VARCHAR(25),
membership_card INTEGER
);
=> INSERT INTO customer2 VALUES (default,'Carr', 'Mary', 87432);
Now query the table you just created. The ID column has been incremented again to 103:
=> SELECT * FROM customer2;
id | lname | fname | membership_card
-----+-------+-------+----------------103 | Carr | Mary |
87432
(1 row)
Administrator's Guide
Managing Tables
(1 row)
As you can see, each time you call NEXTVAL(), the value increments.
The CURRVAL() function returns the current value.
Altering Sequences
The ALTER SEQUENCE statement lets you change the attributes of a previously-defined
sequence. Changes take effect in the next database session. Any parameters not specifically set
in the ALTER SEQUENCE command retain their previous settings.
The ALTER SEQUENCE statement lets you rename an existing sequence, or the schema of a
sequence, but you cannot combine either of these changes with any other optional parameters.
Note: Using ALTER SEQUENCE to set a START value below the CURRVAL can result in
duplicate keys.
Examples
The following example modifies an ascending sequence called my_seq to start at 105:
ALTER SEQUENCE my_seq RESTART WITH 105;
Remember that changes occur only after you start a new database session. For example, if you
create a sequence named my_sequence and start the value at 10, each time you call the NEXTVAL
function, you increment by 1, as in the following series of commands:
CREATE SEQUENCE my_sequence START 10;SELECT NEXTVAL('my_sequence');
nextval
--------10
(1 row)
SELECT NEXTVAL('my_sequence');
nextval
Administrator's Guide
Managing Tables
--------11
(1 row)
Now issue the ALTER SEQUENCE statement to assign a new value starting at 50:
ALTER SEQUENCE my_sequence RESTART WITH 50;
When you call the NEXTVAL function, the sequence is incremented again by 1 value:
SELECT NEXTVAL('my_sequence'); NEXTVAL
--------12
(1 row)
Distributed Sequences
When you create a sequence object, the CACHE parameter controls the sequence efficiency, by
determining the number of sequence values each node maintains during a session. The default
cache value is 250K, meaning that each node reserves 250,000 values for each sequence per
session.
HP Vertica distributes a session across all nodes. After you create a sequence, the first time a
node executes a NEXTVAL() function as part of a SQL statement, the node reserves its own cache
of sequence values. The node then maintains that set of values for the current session. Other
nodes executing a NEXTVAL() function will also create and maintain their own cache of sequence
values cache.
Note: If any node consumes all of its sequence values, HP Vertica must perform a catalog
lock to obtain a new set of values. A catalog lock can be costly in terms of database
performance, since certain activities, such as data inserts, cannot occur until HP Vertica
releases the lock.
During a session, one node can use its allocated set of sequence values slowly, while another node
uses its values more quickly. Therefore, the value returned from NEXTVAL in one statement can
differ from the values returned in another statement executed on another node.
Administrator's Guide
Managing Tables
At the end of every statement, the state of all sequences used in the session is returned to the
initiator node.
The initiator node calculates the maximum CURRVAL of each sequence across all states on all
nodes.
This maximum value is used as CURRVAL in subsequent statements until another NEXTVAL
is invoked.
If a statement fails after NEXTVAL is called (thereby consuming a sequence value from the
cache), the value is lost.
If a disconnect occurs (for example, dropped session), any remaining values in the cache that
have not been returned through NEXTVAL (unused) are lost.
To recover lost sequence values, you can run an ALTER SEQUENCE command to define a new
sequence number generator, which resets the counter to the correct value in the next session.
Note: An elastic projection (a segmented projection created when Elastic Cluster is enabled)
created with a modular hash segmentation expression uses hash instead.
The behavior of sequences across HP Vertica nodes is explained in the following examples.
Note: IDENTITY and AUTO_INCREMENT columns behave in a similar manner.
Example 1: The following example, which illustrates sequence distribution, assumes a 3-node
cluster with node01 as the initiator node.
First create a simple table called dist:
CREATE TABLE dist (i INT, j VARCHAR);
Administrator's Guide
Managing Tables
Create a second projection called twoNodes and segment column x by modularhash on node02 and
node03:
CREATE PROJECTION twoNodes AS SELECT * FROM dist SEGMENTED BY MODULARHASH(i) NODES
node02, node03;
Create a third projection called threeNodes and segment column i by modularhash on all nodes (13):
CREATE PROJECTION threeNodes as SELECT * FROM dist SEGMENTED BY MODULARHASH(i) ALL NODES;
The following table shows the segmentation of rows for projection oneNode:
1
2
3
4
5
6
ONE
TWO
THREE
FOUR
FIVE
SIX
Node01
Node01
Node01
Node01
Node01
Node01
The following table shows the segmentation of rows for projection twoNodes:
Administrator's Guide
Managing Tables
1
2
3
4
5
6
ONE
TWO
THREE
FOUR
FIVE
SIX
Node03
Node02
Node03
Node02
Node03
Node02
The following table shows the segmentation of rows for projection threeNodes:
1
2
3
4
5
6
ONE
TWO
THREE
FOUR
FIVE
SIX
Node02
Node03
Node01
Node02
Node03
Node01
Create a sequence and specify a cache of 10. The sequence will cache up to 10 values in memory
for performance. As per the CREATE SEQUENCE statement, the minimum value is 1 (only one
value can be generated at a time, for example, no cache).
Example 2: Create a sequence named s1 and specify a cache of 10:
CREATE SEQUENCE s1 cache 10;SELECT s1.nextval, s1.currval, s1.nextval, s1.currval, j FROM
oneNode;
nextval | currval | nextval | currval |
j
---------+---------+---------+---------+------1 |
1 |
1 |
1 | ONE
2 |
2 |
2 |
2 | TWO
3 |
3 |
3 |
3 | THREE
4 |
4 |
4 |
4 | FOUR
5 |
5 |
5 |
5 | FIVE
6 |
6 |
6 |
6 | SIX
(6 rows)
The following table illustrates the current state of the sequence for that session. It holds the current
value, values remaining (the difference between the current value (6) and the cache (10)), and
cache activity. There is no cache activity on node02 or node03.
Sequence Cache State Node01 Node02
Node03
Current value
NO CACHE NO CACHE
Remainder
NO CACHE NO CACHE
Administrator's Guide
Managing Tables
6
6
6
6
6
6
(6 rows)
|
|
|
|
|
|
ONE
THREE
FIVE
TWO
FOUR
SIX
Example 4: Now call NEXTVAL from threeNodes. The assumption is that node02 holds the cache
before node03:
SELECT s1.nextval, j from threeNodes;
nextval |
j
---------+------101 | ONE
201 | TWO
7 | THREE
102 | FOUR
202 | FIVE
8 | SIX
(6 rows)
The following table illustrates the sequence cache state with values on node01, node02, and
node03:
Sequence Cache State Node01 Node02 Node03
Current value
102
202
Left
102
202
Left
Administrator's Guide
Managing Tables
103
202
Left
Example 7: The following command calls the current value from twoNodes:
SELECT s1.currval, j FROM twoNodes;
currval |
j
---------+------103 | ONE
103 | TWO
103 | THREE
103 | FOUR
103 | FIVE
103 | SIX
(6 rows)
Example 8: This example assume that node02 holds the cache before node03:
SELECT s1.nextval, j FROM twoNodes;
nextval |
j
---------+------203 | ONE
104 | TWO
204 | THREE
105 | FOUR
205 | FIVE
106 | SIX
(6 rows)
106
205
Left
Example 9: The following command calls the current value from oneNode:
Administrator's Guide
Managing Tables
302
106
205
Left
Example 11: In this example, twoNodes is the outer table and threeNodes is the inner table to a
merge join. threeNodes is resegmented as per twoNodes.
SELECT s1.nextval, j FROM twoNodes JOIN threeNodes ON twoNodes.i = threeNodes.i;
Administrator's Guide
Managing Tables
302
109
208
Left
Example 12: This next example shows how sequences work with buddy projections.
--Same session
DROP TABLE t CASCADE;
CREATE TABLE t (i INT, j varchar(20));
CREATE PROJECTION threeNodes AS SELECT * FROM t
SEGMENTED BY MODULARHASH(i) ALL NODES KSAFE 1;
COPY t FROM STDIN;
1|ONE
2|TWO
3|THREE
4|FOUR
5|FIVE
6|SIX
\.
SELECT node_name, projection_name, total_row_count FROM storage_containers;
node_name | projection_name | total_row_count
-----------+-----------------+----------------node01
| threeNodes_b0
|
2
node03
| threeNodes_b0
|
2
node02
| threeNodes_b0
|
2
node02
| threeNodes_b1
|
2
node01
| threeNodes_b1
|
2
node03
| threeNodes_b1
|
2
(6 rows)
The following function call assumes that node02 is down. It is the same session. Node03 takes up
the work of node02:
SELECT s1.nextval, j FROM t;
nextval |
j
---------+------401 | ONE
402 | TWO
305 | THREE
403 | FOUR
404 | FIVE
306 | SIX
(6 rows)
306
110
404
Left
Administrator's Guide
Managing Tables
Node03
Current value
501
NO CACHE NO CACHE
Left
Example 14:
INSERT INTO t SELECT s1.nextval, 'TWO' FROM twoNodes;
SELECT * FROM t;
i
|
j
-----+------501 | ONE --stored in node01 for oneNode, node02 for twoNodes, node02 for threeNodes
601 | TWO --stored in node01 for oneNode, node03 for twoNodes, node01 for threeNodes
(2 rows)
501
601
NO CACHE
Left
Example 15:
INSERT INTO t select
SELECT * FROM t;
i
|
j
-----+------501 | ONE --stored
601 | TWO --stored
502 | TRE --stored
602 | TRE --stored
in
in
in
in
node01
node01
node01
node01
for
for
for
for
oneNode,
oneNode,
oneNode,
oneNode,
node02
node03
node03
node02
for
for
for
for
twoNodes,
twoNodes,
twoNodes,
twoNodes,
node02
node01
node03
node02
for
for
for
for
threeNodes
threeNodes
threeNodes
threeNodes
Administrator's Guide
Managing Tables
(4 rows)
502
602
NO CACHE
Left
Example 16:
INSERT INTO t SELECT
SELECT * FROM t;
i
|
j
-----+------501 | ONE --stored
601 | TWO --stored
502 | TRE --stored
602 | TRE --stored
602 | ONE --stored
502 | TWO --stored
602 | TRE --stored
(7 rows)
in
in
in
in
in
in
in
node01
node01
node01
node01
node01
node01
node01
for
for
for
for
for
for
for
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
node02
node03
node03
node02
node02
node03
node02
for
for
for
for
for
for
for
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
node02
node01
node03
node02
node02
node03
node02
for
for
for
for
for
for
for
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
node02
node01
node03
node02
node02
node03
node02
for
for
for
for
for
for
for
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
threeNodes
502
602
NO CACHE
Left
Example 17:
INSERT INTO t VALUES
SELECT * FROM t;
i
|
j
-----+------501 | ONE --stored
601 | TWO --stored
502 | TRE --stored
602 | TRE --stored
602 | ONE --stored
502 | TWO --stored
602 | TRE --stored
603 | QUA
(8 rows)
(s1.currval + 1, 'QUA');
in
in
in
in
in
in
in
node01
node01
node01
node01
node01
node01
node01
for
for
for
for
for
for
for
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
oneNode,
node02
node03
node03
node02
node02
node03
node02
for
for
for
for
for
for
for
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
twoNodes,
Administrator's Guide
Managing Tables
502
602
NO CACHE
Left
See Also
l
Sequence Privileges
ALTER SEQUENCE
CREATE TABLE
Column-Constraint
CURRVAL
DROP SEQUENCE
GRANT (Sequence)
NEXTVAL
Loading Sequences
You can use a sequence as part of creating a table. The sequence must already exist, and have
been instantiated using the NEXTVAL statement.
After creating a sequence, you must call the NEXTVAL function at least once in a session to create
a cache for the sequence and its initial value. Subsequently, use NEXTVAL to increment the
sequence. Use the CURRVAL function to get the current value.
The following NEXTVAL function instantiates the newly-created my_seq sequence and sets its first
number:
=> SELECT NEXTVAL('my_seq');
Administrator's Guide
Managing Tables
nextval
--------100
(1 row)
Now query the table you just created. The column named ID has been incremented by (1) again to
104:
SELECT * FROM customer2; ID | lname | fname | membership_card
-----+-------+-------+----------------104 | Carr | Mary |
87432
(1 row)
Dropping Sequences
Use the DROP SEQUENCE function to remove a sequence. You cannot drop a sequence:
l
If other objects depend on the sequence. The CASCADE keyword is not supported.
That is used in the default expression of a column until all references to the sequence are
removed from the default expression.
Administrator's Guide
Managing Tables
Example
The following command drops the sequence named my_sequence:
=> DROP SEQUENCE my_sequence;
A target tableThe master table that contains existing rows you want to update or insert into
using rows from another (source) table.
A source tableThe table that contains the new and/or changed rows you'll use to update the
target table.
A search conditionThe merge join columns, specified in the ON clause, that HP Vertica uses to
evaluate each row in the source table to update, delete, and/or insert rows into the target table.
Additional filters that instruct HP Vertica what to do when the search condition is or is not met.
For example, when you use:
n
WHEN MATCHED THEN UPDATE clause HP Vertica updates and/or deletes existing rows in
the target table with data from the source table.
Note: HP Vertica assumes the values in the merge join column are unique. If more than
one matching value is present in either the target or source table's join column, the MERGE
statement could fail with a run-time error. See Optimized Versus Non-optimized MERGE
for more information.
WHEN NOT MATCHED THEN INSERT clauseHP Vertica inserts into the target table all rows
from the source table that do not match any rows in the target table.
Administrator's Guide
Managing Tables
without optimization or return a run-time error. This section describes scenarios for both optimized
and non-optimized MERGE.
The target table's join column has a unique or primary key constraint
UPDATE and INSERT clauses include every column in the target table
When the above conditions are not met, HP Vertica prepares a non-optimized query plan, and
MERGE runs with the same performance as in HP Vertica 6.1 and prior.
Note: The source table's join column does not require a unique or primary key constraint to be
eligible for an optimized query plan. Also, the source table can contain more columns than the
target table, as long as the UPDATE and INSERT clauses use the same columns and the column
attributes are the same.
How to determine if a MERGE statement is eligible for optimization
To determine whether a MERGE statement is eligible for optimization, prefix MERGE with the EXPLAIN
keyword and examine the plan's textual output. (See Viewing the MERGE Query Plan for
examples.) A a Semi path indicates the statement is eligible for optimization, whereas a Right
Outer path indicates the statement is ineligible and will run with the same performance as MERGE in
previous releases unless a duplicate merge join key is encountered at query run time.
Duplicates in the source table. If HP Vertica finds more than one matching value in the source
join column for a corresponding value in the target table, MERGE fails with a run-time error.
Duplicates in the target table. If HP Vertica finds more than one matching value in target join
column for a corresponding value in the source table, and the target join column has a unique or
Administrator's Guide
Managing Tables
primary key constraint, MERGE fails with a run-time error. If the target join column has no such
constraint, the statement runs without error and without optimization.
Be aware that if you run MERGE multiple times using the same target and source table, each
statement run has the potential to introduce duplicate values into the join columns, such as if you
use constants in the UPDATE/INSERT clauses. These duplicates could cause a run-time error the
next time you run MERGE.
To avoid duplicate key errors, be sure to enforce constraints you declare to assure unique values in
the merge join column.
Examples
The examples that follow use a simple schema to illustrate some of the conditions under which HP
Vertica prepares or does not prepare an optimized query plan for MERGE:
CREATE TABLE target(a INT PRIMARY KEY, b INT, c INT) ORDER BY b,a;
CREATE TABLE source(a INT, b INT, c INT) ORDER BY b,a;
INSERT INTO target VALUES(1,2,3);
INSERT INTO target VALUES(2,4,7);
INSERT INTO source VALUES(3,4,5);
INSERT INTO source VALUES(4,6,9);
COMMIT;
The target table's join column (ON t.a=s.a) has a primary key constraint
All columns in the target table (a,b,c) are included in the UPDATE and INSERT clauses
Columns attributes specified in the UPDATE and INSERT clauses are identical
MERGE INTO target t USING source s ON t.a = s.a
WHEN MATCHED THEN UPDATE SET a=s.a, b=s.b, c=s.c
WHEN NOT MATCHED THEN INSERT(a,b,c) VALUES(s.a,s.b,s.c);
OUTPUT
-------2
(1 row)
The output value of 2 indicates success and denotes the number of rows updated/inserted from the
source into the target.
Example of a non-optimized MERGE statement
Administrator's Guide
Managing Tables
In the next example, the MERGE statement runs without optimization because the column attributes
in the UPDATE/INSERT clauses are not identical. Specifically, the UPDATE clause includes constants
for columns s.a and s.c and the INSERT clause does not:
MERGE INTO target t USING source s ON t.a = s.a
WHEN MATCHED THEN UPDATE SET a=s.a + 1, b=s.b, c=s.c - 1
WHEN NOT MATCHED THEN INSERT(a,b,c) VALUES(s.a,s.b,s.c);
To make the previous MERGE statement eligible for optimization, rewrite the statement as follows,
so the attributes in the UPDATE and INSERT clauses are identical:
MERGE INTO target t USING source s ON t.a = s.a
WHEN MATCHED THEN UPDATE SET a=s.a + 1, b=s.b, c=s.c -1
WHEN NOT MATCHED THEN INSERT(a,b,c)
VALUES(s.a + 1, s.b, s.c - 1);
Administrator's Guide
Managing Tables
Dropping Tables
Dropping a table removes its definition from the HP Vertica database. For the syntax details of this
statement, see DROP TABLE in the SQL Reference Manual.
To drop a table, use the following statement:
=> DROP TABLE IF EXISTS mytable;
DROP TABLE
=> DROP TABLE IF EXISTS mytable; -- Table doesn't exist
NOTICE: Nothing was dropped
DROP TABLE
If the table you specify has dependent objects such as projections, you cannot drop the table. You
must use the CASCADEkeyword. If you use the CASCADEkeyword, HP Vertica drops the table
and all its dependent objects. These objects can include superprojections, live aggregate
projections, and projections with expressions.
You cannot use the CASCADE option to drop an external table. Because an external table is read
only, you cannot remove any of its associated files.
Truncating Tables
Truncating a table removes all storage associated with that table and its projections. HP Vertica
preserves the table and the projection definitions.
You cannot truncate an external table.
Use TRUNCATE TABLE for testing purposes. With this table, you can remove all table data
without having to recreate projections when you reload table data. For details, see TRUNCATE
TABLE in the SQL Reference Manual.
Administrator's Guide
Managing Tables
The TRUNCATE TABLE statement commits the entire transaction after statement execution, even
if truncating the table fails. You cannot roll back a TRUNCATE statement.
In some cases, you might use create truncated table from a large single (fact) table containing prejoin projections.. When you do so, the projections show zero (0) rows after the transaction
completes and the table is ready for data reload.
If the table to truncate is a dimension table with pre-join projections, you cannot truncate it. Drop the
pre-join projections before executing the TRUNCATE TABLE statement.
If the truncated table has out-of-date projections, those projections are cleared and marked up-todate after the TRUNCATE TABLE operation completes.
TRUNCATE TABLE takes an O (Owner) lock on the table until the truncation process completes.
The savepoint is then released.
Administrator's Guide
Using Text Search
Creating a TextIndex
About TxtIndex.Stemmer
Administrator's Guide
Using Text Search
table.
Creating a TextIndex
In the following example, you perform a text search using a source table called t_log. This source
table has two columns:
l
Note: You must associate a projection with the source table. This projection must be sorted by
the primary key and either segmented by hash(id) or unsegmented. You can define this
projection on the source table, along with any other existing projections.
In vsql, create a text index on the table for which you want to perform a text search.
=> CREATE TEXTINDEX text_index ON t_log (id, text);
If your table is partitioned, then your text index also contains a third column called partition.
=> SELECT * FROM text_index;
word
| doc_id | partition
------------------------+--------+----------<info>
|
6 |
2014
<warning>
|
2 |
2014
<warning>
|
3 |
2014
<warning>
|
4 |
2014
<warning>
|
5 |
2014
database
|
6 |
2014
execute:
|
6 |
2014
object
|
4 |
2014
object
|
5 |
2014
[catalog]
|
4 |
2014
[catalog]
|
5 |
2014
Administrator's Guide
Using Text Search
You create a text index on a source table only once. In the future, you will not have to re-create the
text index each time the source table is updated or changed.
Your text index stays synchronized to the contents of the source table through any operation that is
run on the source table. These operations include, but are not limited to:
l
COPY
INSERT
UPDATE
DELETE
DROP PARTITION
MOVE_PARTITION_TO_TABLE
n
When moving or swapping partitions, if the source table is indexed then the destination table
must already exist and be indexed in the same way.
The results show the matched keyword you searched and the associated doc_id, where the
keyword is located.
=> SELECT * FROM text_index WHERE word = TxtIndex.Stemmer(LOWER('<WARNING>'));
word
| doc_id
-----------+-------<warning> |
2
<warning> |
3
<warning> |
4
<warning> |
5
(4 rows)
Next, write a query to display the full contents of the source table that match the keyword you
searched for in the text index.
=> SELECT* FROM t_log WHERE id IN (SELECT doc_id FROM text_index WHERE word =
TxtIndex.Stemmer(LOWER('<WARNING>')));
id |
date
|
text
Administrator's Guide
Using Text Search
Use the doc_id to find the exact location of the keyword in the source table.The doc_id matches
the unique identifier from the source table. This matching allows you to quickly find the instance of
the keyword in your table.
In this example, t_log is the source table, id is the unique id field in the source table, text is the
text field in the source table, and text_index is the text index. The query finds case sensitive
records:
SELECT * FROM t_log WHERE id IN (SELECT doc_id FROM text_index WHERE word =
TxtIndex.Stemmer(LOWER('warning'))) AND text LIKE '%Warning%';
Administrator's Guide
Using Text Search
In this example, t_log is the source table, and text_index is the text index. The query finds
records that either contain:
l
An administrative task needs to be performed on the source table and requires the text index to
be dropped.
Dropping the text index does not drop the source table associated with the text index. However, if
you drop the source table associated with a text index, then that text index is also dropped. The text
index is considered a dependent object.
The following example illustrates how to drop a text index named text_index:
Administrator's Guide
Using Text Search
About TxtIndex.Stemmer
TxtIndex.Stemmer finds words that are similar, but not necessarily identical, to the specified
keyword. For example, if you perform a search on a text index for the keyword "database," you
might also want to get results containing the word "databases."
To achieve this type of matching, HP Vertica represents words in their stem form when using
TxtIndex.Stemmer. In the following example "query" is the stem word.
Suppose you search "query" in a text index that contains query, queries, queried, and querying.
In this case, you get four results because each of the words share the same stem.
=> SELECT * FROM t_log WHERE id IN (SELECT doc_id FROM text_index WHERE word =
TxtIndex.Stemmer(LOWER('query')));
id |
text
----+---------14 | query
18 | queries
19 | queried
11 | querying
(4 rows)
Administrator's Guide
Working withProjections
Working withProjections
Projections are physical storage for table data. A projection can contain some or all of the columns
from one or more tables. The sections that follow explain the types of projections you can create in
an HP Vertica database and how you can use them to optimize query performance.
l
Top-K Projections
Types of Projections
You can create several types of projections.
Superprojections
Asuperprojection contains all the columns of a table. For each table in the database, HP Vertica
requires a minimum of one projection, which is the superprojection.To get your database up and
running quickly, when you load or insert data into an existing table for the first time, HP Vertica
automatically creates a superprojection.
Query-Specific Projections
Aquery-specific projection is a projection that contains only the subset of table columns to process
a given query. Query-specific projections significantly improve the performance of those queries for
which they are optimized.
Pre-Join Projections
Apre-join projection contains inner joins between tables that are connected by PK/FK constraints.
Pre-join projections provide a significant performance advantage over joining tables at query run
time. In addition, using a pre-join projection can enable you to define sort orders for queries that you
execute frequently.
Anchor Projections
An anchor projection is the physical storage for the anchor table data.
Administrator's Guide
Working withProjections
Before you create a live aggregate projection, you must create an anchor projection for the anchor
table. The anchor projection's segmentation must be a subset of the live aggregate projection's
segmentation. Each anchor table used for a live aggregate projection must have at least one anchor
projection.
When you create a Top-K projection, the anchor table must have at least one projection that has the
same PARTITIONBYdefinition as in the CREATEPROJECTIONstatement.This projection is
the anchor projection for the Top-K projection.
Top-K Projections
ATop-Kprojection is a projection that is optimized for Top-K optimization. ATop-K query retrieves
the top k rows from within a group of tuples.
Projection Basics
The following topics provide some basic information you need to know about managing projections
in your database:
l
Administrator's Guide
Working withProjections
However, the refresh does not begin until after you create buddy projection . To monitor the refresh
operation, review the vertica.log file. You can also run SELECT GET_PROJECTIONS('tablename;) to view the final status of the projection refresh:
=> SELECT GET_PROJECTIONS('customer_dimension');
GET_PROJECTIONS
---------------------------------------------------------------Current system K is 1. # of Nodes: 4. Table public.customer_dimension has 4
projections.
Projection Name: [Segmented] [Seg Cols] [# of Buddies] [Buddy Projections]
[Safe] [UptoDate]
---------------------------------------------------------public.customer_dimension_node0004 [Segmented: No] [Seg Cols: ] [K: 3]
[public.customer_dimension_node0003, public.customer_dimension_node0002, public.customer_
dimension_node0001]
[Safe: Yes] [UptoDate: Yes][Stats: Yes]public.customer_dimension_node0003
[Segmented: No] [Seg Cols: ] [K: 3] [public.customer_dimension_node0004, public.customer_
dimension_node0002, public.customer_dimension_node0001]
[Safe: Yes] [UptoDate: Yes][Stats: Yes]public.customer_dimension_node0002
[Segmented: No] [Seg Cols: ] [K: 3] [public.customer_dimension_node0004, public.customer_
dimension_node0003, public.customer_dimension_node0001]
[Safe: Yes] [UptoDate: Yes][Stats: Yes]public.customer_dimension_node0001
[Segmented: No] [Seg Cols: ] [K: 3] [public.customer_dimension_node0004, public.customer_
dimension_node0003, public.customer_dimension_node0002]
[Safe: Yes] [UptoDate: Yes][Stats: Yes](1 row)
Administrator's Guide
Working withProjections
For example, suppose that the cluster node names are NODE01, NODE02, and NODE03. When
you create an unsegmented projection with the following command, you also create projections
named ABC_NODE01, ABC_NODE02, and ABC_NODE03:
=> CREATE PROJECTION ABC ... UNSEGMENTED ALL NODES;
Creating unsegmented projections affects the projection name argument value for functions like
GET_PROJECTIONS or GET_PROJECTION_STATUS. For these functions, you must provide
the entire projection name, including the node-name suffix (ABC_NODE01), rather than the projection
name alone (ABC).
Live aggregate projectionsProjections that contain columns whose values have been
aggregated from columns in its anchor table. For details, see Live Aggregate Projections.
Top-K projections A type of live aggregate projection that returns the top k rows from a
partition of selected rows. Create a Top-K projection that satisfies the criteria for a Top-K query.
For details, see Top-K Projections.
Projections that contain expressions that calculate projection column values from anchor table
columnsThese projections behave similarly to normal projections, with a few exceptions. If
the expressions perform aggregate operations that live aggregate projections support, such as
SUM(), HP Vertica treats them like live aggregate projections. For details, see Projections with
Expressions
Administrator's Guide
Working withProjections
Detect fraud.
HP Vertica can analyze this data through queries that aggregate the data. However, for faster
response, you can create a projection where the data is stored after being aggregated.
This type of projection is a live aggregate projection. The following example describes a Top-K
projection, which is a type of live aggregate projection.
Before you create a live aggregate projection, you must create a projection that receives new data
on a regular basis. This projection is called an anchor projection. The data in the anchor projection
is not aggregated. You cannot perform any update, delete, or merge operations on the anchor
projection. Create an anchor projection whose segmentation will match the segmentation of the live
aggregate projection.
In the following graphic, the readings projection stores all meter readings for gas meters.
readings is the anchor projection for the current_readings live aggregate projection. The
current_readings projection contains only the most recent reading for each meter id. When new
readings are recorded, HP Vertica appends the data to the anchor projection. In addition, HP
Vertica aggregates the data and adds the aggregated data to the live aggregate projection at the
appropriate location.
Because new reading for meter id 1232 is older (08:05) than the reading in the live aggregate
projection (08:10), the new reading is ignored.
Administrator's Guide
Working withProjections
For more information about anchor projections, see Anchor Tables and Anchor Projections.
Administrator's Guide
Working withProjections
For the following operations, HP Vertica treats live aggregate projections the same as other types
of projections:
l
Backup
Restore
Refresh
Rebalance
Recovery
Create the anchor projection as follows. This anchor projection is segmented on the user_id
column. The KSAFEkeyword works as follows:
l
If you specify a value less than the system K-safety or greater than the anchor projection Ksafety, an error occurs.
If you specify a value between the system K-safety and the anchor projection K-safety, HP
Vertica sets the live aggregate projection K-safety to the anchor projection K-safety.
If you omit a value for the KSAFEkeyword, HP Vertica sets the live aggregate projection Ksafety to the system K-safety.
=>CREATE PROJECTION clicks_anchorp AS
SELECT * FROM clicks SEGMENTED BY HASH(user_id) ALL NODES KSAFE;
Important: You must create an anchor projection before you create a live aggregate projection.
After an anchor projection has been associated with a live aggregate projection, you cannot
perform any delete, update, or merge operations on the anchor projection.The anchor projection
must specify a segmentation that is a subset of the live aggregate projection's segmentation.
The following example shows how to create a live aggregate projection that counts the number of
clicks per user:
Administrator's Guide
Working withProjections
After the data is loaded, query data directly from the live aggregate projection.
For more detail about creating this example, see Creating Live Aggregate Projections .
You need to be able to update, delete, or merge data in the anchor table.
Before you create a live aggregate projection, you must create an anchor projection. The
anchor projection's segmentation must be a subset of the live aggregate projection's
segmentation and cannot use range segmentation. The anchor table cannot be unsegmented.
After you create a live aggregate projection for a table, you can no longer delete, update, or
merge data in that anchor table. In addition, you cannot change any metadata for that table such
as table name, table definition, or column definition.
Administrator's Guide
Working withProjections
You can only use the aggregate functions listed in Supported Aggregations. You cannot use
aggregate expression with mathematical operators. You can use mathematical operators in
Projections with Expressions, but not in conjunction with aggregate functions.
The list of columns in the SELECTlist for a live aggregate projection must be in the same order
as the list of columns in the GROUPBYclause in the live aggregate projection. The
GROUPBYcolumn must appear at the beginning of the SELECTlist of a live aggregate
projection.
If you use an expression in the PARTITIONBYclause for a live aggregate projection, the anchor
projection segmentation must use the same expression.
You cannot create a live aggregate projection by selecting columns from another projection.
You cannot use an ORDER BYclause when creating a live aggregate projection.
You cannot use the OFFSETclause when creating a live aggregate projection.
You cannot run the following functions on anchor tables of live aggregate projections:
n
PARTITION_TABLE
ANALYZE_STATISTICS. You do not see any error when you run ANALYZE_
STATISTICSon a live aggregate projection, but it does not perform any operations.
PURGE_PROJECTION
ANALYZE_HISTOGRAM. You do not see any error when you run ANALYZE_
HISTOGRAMon a live aggregate projection, but it does not perform any operations.
Database Designer does not recommend live aggregate projections when it creates a design.
Administrator's Guide
Working withProjections
Supported Aggregations
HP Vertica supports the following aggregations for live aggregate projections:
l
SUM [Aggregate]
MAX [Aggregate]
MIN [Aggregate]
COUNT [Aggregate]
Top-K Projections
ATop-K query retrieves the top k rows from partition of selected rows. For example, in the following
Top-K query, table1 has two columns:a and b. The query returns the top 10 rows associated with
each value of column a.
=> SELECT * FROM table1 LIMIT 10 OVER (PARTITIONBY a ORDERBY b);
Backup
Restore
Refresh
Rebalance
Recovery
Administrator's Guide
Working withProjections
For example, suppose you have a table with the following definition. The columns represent a gas
meter id, the time the meter was read, and the value that was read at that time:
=> CREATETABLEreadings (
meter_id INT,
reading_date TIMESTAMP,
reading_value FLOAT);
Create an anchor projection as follows. The anchor projection is segmented on the meter_id
column. The KSAFEkeyword works as follows:
l
If you specify a value less than the system K-safety or greater than the anchor projection Ksafety, you see an error.
If you specify a value between the system K-safety and the anchor projection K-safety, HP
Vertica sets the Top-K projection to the anchor projection K-safety.
If you omit a value for the KSAFEkeyword, HP Vertica sets the Top-K projection K-safety to the
system K-safety.
=>CREATE PROJECTION readings_anchorp (
meter_id,
reading_date,
reading_value)
AS SELECT * FROM readings SEGMENTED BY HASH(meter_id) ALL NODES KSAFE;
Important: You must create an anchor projection before you create a live aggregate projection.
After an anchor projection has been associated with a live aggregate projection, you cannot
perform any delete, update, or merge operations on the anchor projection.
The anchor projection must specify a segmentation that is a subset of the live aggregate
projection's segmentation.
To retrieve the five most recent meter readings for each gas meter, you might enter the following:
=>SELECT meter_id, reading_date, reading_value FROM (
SELECT meter_id, reading_date, reading_value, ROW_NUMBER()
OVER (PARTITION BY meter_id ORDER BY reading_date DESC) rn FROM readings) sq
WHERE rn <= 5;
However, you cannot use subqueries when defining a Top-K projection, so HP Vertica supports a
simplified Top-K syntax:
=> SELECT meter_id, reading_date, reading_value FROM readings LIMIT 5
OVER (PARTITION BY meter_id ORDER BY reading_date DESC);
Administrator's Guide
Working withProjections
Important: You can use either form of the syntax in a Top-K query, but you can only use the
simplified syntax to create a Top-Kprojection. When you use the simplified syntax to query a
table, you do not have the same restrictions as when you use the simplified syntax when
creating a Top-K projection.
Use this query to create a Top-K projection that is organized by meter_id and stores the five most
recent meter readings for each gas meter:
=>CREATEPROJECTIONreadings_topk (meter_id, recent_date, recent_value)
AS
SELECT meter_id, reading_date, reading_value FROM readings
LIMIT 5 OVER (PARTITION BY meter_id ORDER BY reading_date DESC);
After the data is loaded, you can query data directly from the Top-K projection:
=> SELECT *FROMreadings_topk;
Top-Kprojections must include LIMIT, OVER, PARTITION BY, and ORDER BY clauses.
When creating a Top-K projection, you can only use an ORDERBYclause on the
PARTITIONBYclause inside an OVER()clause.
The columns in the PARTITION BY and ORDER BY clauses must also be in the SELECT list.
You cannot use a PARTITION AUTO clause when creating a Top-K projection.
Administrator's Guide
Working withProjections
Stock symbol
Important: You must create an anchor projection before you create a live aggregate
projection. After an anchor projection has been associated with a live aggregate projection,
you cannot perform any delete, update, or merge operations on the anchor projection.
The anchor projection must specify a segmentation that is a subset of the live aggregate
projection's segmentation.
3. Add some data to the table:
INSERT /*+direct*/ INTO trades
VALUES('AAPL','2010-10-10 10:10:10'::TIMESTAMP,100.00,100);
INSERT /*+direct*/ INTO trades
VALUES('AAPL','2010-10-10 10:10:10'::TIMESTAMP,100.00,100);
INSERT /*+direct*/ INTO trades
VALUES ('AAPL','2011-10-10 10:10:10.5'::TIMESTAMP,100.1,1000);
INSERT /*+direct*/ INTO trades
VALUES ('AAPL','2011-10-10 10:10:10.2'::TIMESTAMP,100.2,500);
INSERT /*+direct*/ INTO trades
VALUES ('HPQ','2012-10-10 10:10:10.2'::TIMESTAMP,42.01,300);
INSERT /*+direct*/ INTO trades
VALUES ('HPQ','2012-10-10 10:10:10.2'::TIMESTAMP,42.02,300);
INSERT /*+direct*/ INTO trades
Administrator's Guide
Working withProjections
4. Create a Top-K projection that returns the most recent trades for both stock symbols:
=> CREATE PROJECTION trades_topk_a AS
SELECT symbol, trade_time last_trade, price last_price
FROM trades LIMIT 1 OVER(PARTITION BY symbol ORDER BY trade_time DESC);
=> SELECT * FROM trades_topk_a;
symbol
|
last_trade
| last_price
------------------+-----------------------+-----------HPQ
| 2012-10-10 10:10:10.3 |
42.0500
AAPL
| 2011-10-10 10:10:10.5 |
100.1000
(2 rows)
5. Create a second Top-K projection that calculates the volume-weighted average price (VWAP)
and returns the top VWAPvalues for both stock symbols:
=>CREATE PROJECTION trades_topk_b AS SELECT symbol, price*volume VWAP
FROM trades LIMIT 1 OVER(PARTITION BY symbol ORDER BY price*volume DESC);
=> SELECT * FROM trades_topk_b;
symbol
|
VWAP
------------------+------------AAPL
| 100100.0000
HPQ
| 12615.0000
(2 rows)
6. Create a third Top-Kprojection that returns the most recent trades for stock symbol and trade
time (but not trade date):
=> CREATE PROJECTION trades_topk_c AS SELECT symbol, trade_time::DATE
trade_date, trade_time, price close_price, volume FROM trades
LIMIT 1 OVER(PARTITION BY symbol, trade_time::DATE ORDER BY trade_time DESC);
=> SELECT * FROM trades_topk_c;
symbol
| trade_date |
trade_time
| close_price | volume
------------------+------------+-----------------------+-------------+-------AAPL
| 2010-10-10 | 2010-10-10 10:10:10
|
100.0000 |
100
HPQ
| 2012-10-10 | 2012-10-10 10:10:10.3 |
42.0500 |
300
AAPL
| 2011-10-10 | 2011-10-10 10:10:10.5 |
100.1000 |
1000
(3 rows)
Administrator's Guide
Working withProjections
anchor table. For the following operations, HP Vertica treats projections with expressions the same
as other types of projections:
l
Backup
Restore
Refresh
Rebalance
Recovery
When you load data into a projection with expressions, HP Vertica automatically calculates the
values according to the expressions and inserts the data into that new projection.
Query the calculated data directly from the projection instead of querying the data from the anchor
table and then calculating the values. This approach eliminates resource-intensive computations
because HP Vertica performs the calculations when it loads the data, rather than each time you
query the projection.
Using expressions in projections also provides the opportunity to sort or segment the data based on
the calculated results of an expression instead of sorting by single column values.
Suppose you want to calculate the product of the two values, as follows:
=>SELECTa, b, a*b FROMvalues;
To create a projection that calculates the product of a and b, use a statement like the following:
=>CREATE PROJECTION values_product (a, b, product_value)
AS SELECT a, b, a*b FROM values
SEGMENTED BY HASH(a) ALL NODES KSAFE;
To query that projection, you must use the name that HP Vertica assigned to it or its buddy
projections:
=>SELECT* FROMvalues_product_b0;
Administrator's Guide
Working withProjections
or
=>SELECT* FROMvalues_product_b1;
To see how the sales vary by month of the year, add up sales for every day of the month:
=> SELECT MONTH(date) AS month, SUM(sales) AS
total FROM monthly_sales GROUP BY MONTH(date);
This query uses the MONTHfunction to extract the month value from the date column. It also uses
the aggregate function SUM, qualifying the projection as a live aggregate projection. Before you
create the projection, create an anchor projection as follows:
=> CREATEPROJECTIONsales_anchorp AS
SELECT *FROMmonthly_sales SEGMENTED BY HASH(MONTH(date)) ALL NODES KSAFE1;
Important: You must create an anchor projection before you create a live aggregate projection.
After an anchor projection has been associated with a live aggregate projection, you cannot
perform any delete, update, or merge operations on the anchor projection.
The anchor projection must specify a segmentation that is a subset of the live aggregate
projection's segmentation.
Rather than re-extracting the month and recalculating the sales total every time you run this query,
create the following projection. HP Vertica automatically calculates the monthly sales each time a
record is added to the monthly_sales anchor table:
=>CREATE PROJECTION sales_by_month (month, total_sales) AS
SELECT MONTH(date) AS month, SUM(sales) AS total
FROM monthly_sales GROUP BY MONTH(date);
sales_by_month is also a live aggregate expression because it aggregates the data using the
SUMfunction with the sales column.
Administrator's Guide
Working withProjections
After you create a projection with expressions but not aggregate expressions, you can delete
and update data in the anchor table, but you cannot perform a merge operation.
Expressions used in projections must be immutable; they must always return the same result.
Two examples of expressions you cannot use are:
n
The expression can reference one or more columns from the table.
Administrator's Guide
Working withProjections
Important: You must create an anchor projection before you create a live aggregate
projection. After an anchor projection has been associated with a live aggregate projection,
you cannot perform any delete, update, or merge operations on the anchor projection.
The anchor projection must specify a segmentation that is a subset of the live aggregate
projection's segmentation.
4. Load data into the anchor table:
=> INSERT /*+direct*/ INTO clicks
VALUES(1, 10001, '2010-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(1, 10001, '2010-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(2, 2002, '2011-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(2, 2002, '2011-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(3, 3003, '2012-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(3, 3003, '2012-10-10 10:10:10'::TIMESTAMP);
INSERT /*+direct*/ INTO clicks
VALUES(3, 3003, '2012-10-10 10:10:10'::TIMESTAMP);
=> COMMIT;
5. Suppose the query that you run frequently counts the number of clicks per user per web page:
=> SELECT user_id, page_id, click_time::DATE click_date,
COUNT(*) num_clicks FROM clicks
GROUP BY user_id, page_id, click_time::DATE;
6. Create a live aggregate projection named clicks_agg that stores the data aggregated as
specified in the previous query:
=> CREATE PROJECTION clicks_agg
AS SELECT user_id, page_id, click_time::DATE click_date,
COUNT(*) num_clicks FROM clicks
GROUP BY user_id, page_id, click_time::DATE;
Administrator's Guide
Working withProjections
8. Test your new projection to verify that you see the expected results:
=>SELECT*FROMclicks_agg;
9. To specify a display order for the results of querying the live aggregate projection, add the
ORDERBYclause to the query.
You cannot use the ORDERBYclause when you create a live aggregate projection. Internally,
the data is ordered on the GROUPBY column. However, when you retrieve the data, the
results do not necessarily display in that order. Use the ORDERBYclause to sort the results
of querying the live aggregate projection:
=>SELECT*FROMclicks_agg ORDERBYuser_id;
Suppose the readings table from the example in Top-K Projections has 59 rows, which includes
readings for five meter ids.
To retrieve the five most recent meter readings for each gas meter, you use a Top-K query to select
the data directly from the readings anchor table. In this example, HP Vertica has to read all 59
rows from the readings table:
=> SELECT meter_id, reading_date, reading_value
FROM readings
LIMIT 5 OVER (PARTITION BY meter_id ORDER BY reading_date DESC);
Administrator's Guide
Working withProjections
In executing the following query, HP Vertica only has to read 25 rows, because the Top-K projection
only stores the five most recent readings for the five meter ids for which it contains data:
=> SELECT * FROM readings_topk;
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+DIRECT*/
/*+direct*/
/*+direct*/
/*+direct*/
/*+direct*/
/*+direct*/
/*+direct*/
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
INTO
values
values
values
values
values
values
values
values
values
values
values
values
values
values
values
values
values
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
VALUES
(34, 100);
(3, 55);
(31, 43);
(56, 434);
(22, 111);
(3, 11);
(4, 2);
(16, 41);
(13, 2);
(8, 23);
(6, 9);
(71, 4);
(33, 8);
(10, 10);
(5, 15);
(11, 11);
(55, 6);
Administrator's Guide
Working withProjections
4. To calculate the product of the two columns, you might use a query like the following. Every
time you run the query, HP Vertica has to recalculate the values:
=>SELECTa, b, a*b as product from values;
5. Instead, create a projection with an expression that stores the calculated values. Every time
the anchor table is updated, HP Vertica updates the projection:
=>CREATE PROJECTION values_product (a, b, product) AS SELECT a, b, a*b FROM values
SEGMENTED BY HASH(a) ALL NODES KSAFE;
6. When you query the projection, HP Vertica does not need to calculate the products. Make sure
to query the name that HP Vertica assigns to the projection and its buddy:
=>SELECT *FROMvalues_product_b0;
=>SELECT*FROMvalues_product_b1;
The following simple table contains two INTEGERvalues whose product you want to calculate:
=>CREATETABLEvalues (a INT, b INT);
To calculate the product, you might use a query like the following:
=>SELECTa, b, a*b ASproduct_value FROMvalues;
To create a projection that calculates the product of a and b, use a statement like the following:
=>CREATE PROJECTION values_product (a, b, product_value)
Administrator's Guide
Working withProjections
If you query the projection, you do not have to recalculate the products each time. The product
values are stored in the projection. To query that projection, you must use the name that HP Vertica
assigned to the projection or one of its buddy projections:
=>SELECT* FROMvalues_product_b0;
or
=>SELECT* FROMvalues_product_b1;
Field Name
DataType
Description
TABLES
HAS_AGGREGATE_
PROJECTION
BOOLEAN
Administrator's Guide
Working withProjections
Field Name
DataType
Description
PROJECTIONS
HAS_EXPRESSIONS
BOOLEAN
IS_AGGREGATE_
PROJECTION
BOOLEAN
AGGREGATE_TYPE
VARCHAR
GROUPBY
TOPK
Administrator's Guide
Working withProjections
Field Name
DataType
Description
PROJECTION_
IS_EXPRESSION
BOOLEAN
COLUMNS
are calculated by an
expression.
IS_AGGREGATE
BOOLEAN
PARTITION_BY_
POSITION
INTEGER
ORDER_BY_POSITION
INTEGER
ORDER_BY_TYPE
INTEGER
COLUMN_EXPRESSION
VARCHAR
ASCNULLSFIRST
ASCNULLSLAST
DESCNULLSFIRST
DESCNULLSLAST
Administrator's Guide
Working withProjections
Administrator's Guide
About Constraints
About Constraints
Constraints specify rules on data that can go into a column. Some examples of constraints are:
l
Uniqueness
Not NULL
Default values
Use constraints when you want to ensure the integrity of your data in one or more columns, but be
aware that it is your responsibility to ensure data integrity. HP Vertica can use constraints to
perform optimizations (such as the optimized MERGE) that assume the data is consistent. Do not
define constraints on columns unless you expect to keep the data consistent.
Administrator's Guide
About Constraints
Adding Constraints
Add constraints on one or more table columns using the following SQL commands:
l
There are two syntax definitions you can use to add or change a constraint:
l
table-constraintUse this syntax when you add a constraint after a column definition in a
CREATE TABLE statement, or when you add, alter, or drop a constraint on a column using
ALTER TABLE.
HP Vertica recommends naming a constraint but it is optional; if you specify the CONSTRAINT
keyword, you must give a name for the constraint.
The examples that follow illustrate several ways of adding constraints. For additional details, see:
l
Unique Constraints
On the column definition using the CONSTRAINT keyword, which requires that you assign a
constraint name, in this example, dim1PK:
CREATE TABLE dim1 ( c1 INTEGER CONSTRAINT dim1PK PRIMARY KEY,
c2 INTEGER
);
On the column definition, omitting the CONSTRAINT keyword. When you omit the
Administrator's Guide
About Constraints
After the column definition, using the CONSTRAINT keyword and assigning a name, in this
example, dim1PK:
CREATE TABLE dim1 ( c1 INTEGER,
c2 INTEGER,
CONSTRAINT dim1pk PRIMARY KEY(c1)
);
On the column definition, use the CONSTRAINT and REFERENCES keywords and name the
constraint, in this example, fact1dim1PK. This example creates a column with a named foreign
key constraint referencing the table (dim1) with the primary key (c1):
Administrator's Guide
About Constraints
On the column definition, omit the CONSTRAINT keyword and use the REFERENCES
After the column definition, use the CONSTRAINT, FOREIGN KEY, and REFERENCES
Each of the following ALTER TABLE statements adds a foreign key constraint on an existing
column, with and without using the CONSTRAINT keyword:
ALTER TABLE fact2
ADD CONSTRAINT fk1 FOREIGN KEY (c1) REFERENCES dim2(c1);
or
ALTER TABLE fact2
Administrator's Guide
About Constraints
To specify multicolumn (compound) primary keys, the following example uses CREATE TABLE to
define the columns. After creating the table, ALTER TABLE defines the compound primary key and
names it dim2PK:
CREATE TABLE dim2 ( c1 INTEGER,
c2 INTEGER,
c3 INTEGER NOT NULL,
c4 INTEGER UNIQUE
);
ALTER TABLE dim2
ADD CONSTRAINT dim2PK PRIMARY KEY (c1, c2);
In the next example, you define a compound primary key as part of the CREATE TABLE
statement. Then you specify the matching foreign key constraint to table dim2 using CREATE
TABLE and ALTER TABLE:
CREATE TABLE dim2 ( c1 INTEGER,
c2 INTEGER,
c3 INTEGER NOT NULL,
c4 INTEGER UNIQUE,
PRIMARY KEY (c1, c2)
);
CREATE TABLE fact2 (
c1 INTEGER,
c2 INTEGER,
c3 INTEGER NOT NULL,
c4 INTEGER UNIQUE
);
ALTER TABLE fact2
ADD CONSTRAINT fact2FK FOREIGN KEY (c1, c2) REFERENCES dim2(c1, c2);
Specify a foreign key constraint using a reference to the table that contains the primary key. In the
ADD CONSTRAINT clause, the REFERENCES column names are optional. The following
ALTER TABLE statement is equivalent to the previous ALTER TABLE statement:
ALTER TABLE fact2ADD CONSTRAINT fact2FK FOREIGN KEY (c1, c2) REFERENCES dim2;
Administrator's Guide
About Constraints
Use ANALYZE_CONSTRAINTS to check for constraint violations in your column. If you find
violations, use the ALTER COLUMN SET/DROP parameters of the ALTER TABLE statement to
apply or remove a constraint on an existing column.
Use ALTER TABLE with the ALTER COLUMN and SET NOT NULL clauses to add the constraint
on column a in table test6:
ALTER TABLE test6 ALTER COLUMN a SET NOT NULL;
Use the SET NOT NULL or DROP NOT NULL clause to add or remove a not NULL column
constraint:
=> ALTER TABLE T1 ALTER COLUMN x SET NOT NULL;
=> ALTER TABLE T1 ALTER COLUMN x DROP NOT NULL;
Use these clauses to ensure that the column has the proper constraints when you have added or
removed a primary key constraint on a column, or any time you want to add or remove the not NULL
constraint.
Note: A PRIMARY KEY constraint includes a not NULL constraint, but if you drop the
PRIMARY KEY constraint on a column, the not NULL constraint remains on that column.
Enforcing Constraints
To maximize query performance, HP Vertica checks for primary key and foreign key violations
when loading into the fact table of a pre-join projection. For more details, see Enforcing Primary
Key and Foreign Key Constraints.
HP Vertica checks for not NULL constraint violations when loading data, but it does not check for
unique constraint violations.
To enforce constraints, load data without committing it using the COPY with NO COMMIT option
and then perform a post-load check using the ANALYZE_CONSTRAINTS function. If constraint
Administrator's Guide
About Constraints
violations are found, you can roll back the load because you have not committed it. For more
details, see Detecting Constraint Violations.
See Also
l
ALTER TABLE
CREATE TABLE
COPY
ANALYZE_CONSTRAINTS
Alternatively, you can add a primary key constraint after the column is created:
CREATE TABLE employees ( employee_id INTEGER
);
ALTER TABLE employees
ADD PRIMARY KEY (employee_id);
Note: If you specify a primary key constraint using ALTER TABLE, the system returns the
following message, which is informational only. The primary key constraint is added to the
designated column.
Administrator's Guide
About Constraints
Examples
Create a table called inventory to store inventory data:
CREATE TABLE inventory (
date_key INTEGER NOT NULL,
product_key INTEGER NOT NULL,
warehouse_key INTEGER NOT NULL,
...
);
To ensure referential integrity between the inventory and warehouse tables, define a foreign key
constraint called fk_inventory_warehouse on the inventory table that references the warehouse
table:
ALTER TABLE inventory
Administrator's Guide
About Constraints
REFERENCES warehouse(warehouse_key);
In this example, the inventory table is the referencing table and the warehouse table is the
referenced table.
You can also create the foreign key constraint in the CREATE TABLE statement that creates the
inventory table, eliminating the need for the ALTER TABLE statement. If you do not specify one
or more columns, the PRIMARY KEY of the referenced table is used:
CREATE TABLE iventory (date_key INTEGER NOT NULL, product_key INTEGER NOT NULL,
warehouse_key INTEGER NOT NULL REFERENCES warehouse
(warehouse_key),
...
);
A foreign key can also constrain and reference multiple columns. The following example uses
CREATE TABLE to add a foreign key constraint to a pair of columns:
CREATE TABLE t1 ( c1 INTEGER PRIMARY KEY,
c2 INTEGER,
c3 INTEGER,
FOREIGN KEY (c2, c3) REFERENCES other_table (c1, c2)
);
The following two examples use ALTER TABLE to add a foreign key constraint to a pair of
columns. When you use the CONSTRAINT keyword, you must specify a constraint name:
ALTER
ADD
ALTER
ADD
TABLE t
FOREIGN KEY (a, b) REFERENCES other_table(c, d);
TABLE t
CONSTRAINT fk_cname FOREIGN KEY (a, b) REFERENCES other_table(c, d);
Note: The FOREIGN KEY keywords are valid only after the column definition, not on the
column definition.
Unique Constraints
Unique constraints ensure that the data contained in a column or a group of columns is unique with
respect to all rows in the table.
Note: If you add a unique constraint to a column and then insert data into that column that is
not unique with respect to other values in that column, HP Vertica inserts the data anyway. If
your data does not conform to the declared constraints, your queries could yield unexpected
Administrator's Guide
About Constraints
You can specify the constraint after the column definition, with and without naming it:
CREATE TABLE product ( product_key INTEGER NOT NULL,
...,
CONSTRAINT product_key_uk UNIQUE (product_key)
);
CREATE TABLE product (
product_key INTEGER NOT NULL,
...,
UNIQUE (product_key)
);
You can also use ALTER TABLE to specify a unique constraint. This example names the
constraint product_key_UK:
ALTER TABLE product
You can use CREATE TABLE and ALTER TABLE to specify unique constraints on multiple
columns. If a unique constraint refers to a group of columns, separate the column names using
commas. The column listing specifies that the combination of values in the indicated columns is
unique across the whole table, though any one of the columns need not be (and ordinarily isn't)
unique:
CREATE TABLE dim1 (
c2 INTEGER,
c3 INTEGER,
UNIQUE (c1, c2)
);
c1 INTEGER,
Administrator's Guide
About Constraints
The not NULL constraint is implicitly applied to a column when you add the PRIMARY KEY (PK)
constraint. When you designate a column as a primary key, you do not need to specify the not
NULL constraint.
However, if you remove the primary key constraint, the not NULL constraint still applies to the
column. Use the ALTER COLUMN x DROP NOT NULL parameter of the ALTER TABLE
statement to drop the not NULL constraint after dropping the primary key constraint.
The following statement enforces a not NULL constraint on the customer_key column, specifying
that the column cannot accept NULL values.
CREATE TABLE customer (
...
);
Administrator's Guide
About Constraints
Dropping Constraints
To drop named constraints, use the ALTER TABLE command.
The following example drops the constraint factfk2:
=> ALTER TABLE fact2 DROP CONSTRAINT fact2fk;
To drop constraints that you did not assign a name to, query the system table TABLE_
CONSTRAINTS, which returns both system-generated and user-named constraint names:
=> SELECT * FROM TABLE_CONSTRAINTS;
If you do not specify a constraint name, HP Vertica assigns a constraint name that is unique to that
table. In the following output, note the system-generated constraint name C_PRIMARY and the userdefined constraint name fk_inventory_date:
-[RECORD 1 ]--------+--------------------------constraint_id
constraint_name
| C_PRIMARY
constraint_schema_id | 45035996273704966
constraint_key_count | 1
foreign_key_count
| 0
table_id
| 45035996273707982
foreign_table_id
| 0
constraint_type
| p
-[ ...
]---------+--------------------------[RECORD 9 ]--------+-------------------------constraint_id
| 45035996273708016
constraint_name
| fk_inventory_date
constraint_schema_id | 0
constraint_key_count | 1
foreign_key_count
| 1
table_id
| 45035996273708014
foreign_table_id
| 45035996273707994
constraint_type
| f
| 45035996273707984
Once you know the name of the constraint, you can then drop it using the ALTER TABLE
command. (If you do not know the table name, use table_id to retrieve table_name from the ALL_
TABLES table.)
Notes
l
Primary key constraints cannot be dropped if there is another table with a foreign key constraint
that references the primary key.
A foreign key constraint cannot be dropped if there are any pre-join projections on the table.
Administrator's Guide
About Constraints
Dropping a primary or foreign key constraint does not automatically drop the not NULL constraint
on a column. You need to manually drop this constraint if you no longer want it.
See Also
l
ALTER TABLE
Administrator's Guide
About Constraints
An outer join is treated as an inner join due to the presence of foreign key.
A new pre-join projection anchored on the table with the foreign key constraint is refreshed.
Administrator's Guide
About Constraints
Pass an empty argument to check for violations on all tables within the current schema
Pass a single table argument to check for violations on the specified table
Pass two arguments, a table name and a column or list of columns, to check for violations in
those columns
Given the following inputs, HP Vertica returns one row, indicating one violation, because the same
primary key value (10) was inserted into table t1 twice:
CREATE TABLE t1(c1 INT);
ALTER TABLE t1 ADD CONSTRAINT pk_t1 PRIMARY KEY (c1);
CREATE PROJECTION t1_p (c1) AS SELECT *
FROM t1 UNSEGMENTED ALL NODES;
INSERT INTO t1 values (10);
INSERT INTO t1 values (10); --Duplicate primary key value
\x
Expanded display is on.
SELECT ANALYZE_CONSTRAINTS('t1');
-[RECORD 1 ]---+-------Schema Name
| public
Table Name
| t1
Column Names
| c1
Constraint Name | pk_t1
Constraint Type | PRIMARY
Column Values
| ('10')
If the second INSERT statement above had contained any different value, the result would have
been 0 rows (no violations).
In the following example, create a table that contains three integer columns, one a unique key and
one a primary key:
CREATE TABLE table_1(
a INTEGER,
b_UK INTEGER UNIQUE,
c_PK INTEGER PRIMARY KEY
);
Administrator's Guide
About Constraints
SELECT ANALYZE_CONSTRAINTS('a_BB');
ERROR: 'a_BB' is not a table name in the current search path
Insert some values into table table_1 and commit the changes:
INSERT INTO table_1 values (1, 1, 1);
COMMIT;
Insert duplicate unique and primary key values and run ANALYZE_CONSTRAINTS on table
table_1 again. The system shows two violations: one against the primary key and one against the
unique key:
INSERT INTO table_1 VALUES (1, 1, 1);
COMMIT;
SELECT ANALYZE_CONSTRAINTS('table_1');
The following command looks for constraint validations on only the unique key in the table table_1,
qualified with its schema name:
=> SELECT ANALYZE_CONSTRAINTS('public.table_1', 'b_UK');
-[RECORD 1 ]---+---------
Administrator's Guide
About Constraints
Schema Name
Table Name
Column Names
Constraint Name
Constraint Type
Column Values
|
|
|
|
|
|
public
table_1
b_UK
C_UNIQUE
UNIQUE
('1')
(1 row)
The following example shows that you can specify the same column more than once; ANALYZE_
CONSTRAINTS, however, returns the violation only once:
SELECT ANALYZE_CONSTRAINTS('table_1', 'c_PK, C_PK');
-[RECORD 1 ]---+---------Schema Name
| public
Table Name
| table_1
Column Names
| c_PK
Constraint Name | C_PRIMARY
Constraint Type | PRIMARY
Column Values
| ('1')
The following example creates a new table, table_2, and inserts a foreign key and different
(character) data types:
CREATE TABLE table_2 (
x VARCHAR(3),
y_PK VARCHAR(4),
z_FK INTEGER REFERENCES table_1(c_PK));
Alter the table to create a multicolumn unique key and multicolumn foreign key and create
superprojections:
ALTER TABLE table_2
ADD CONSTRAINT table_2_multiuk PRIMARY KEY (x, y_PK);
WARNING 2623: Column "x" definition changed to NOT NULL
WARNING 2623: Column "y_PK" definition changed to NOT NULL
The following command inserts a missing foreign key (0) into table dim_1 and commits the
changes:
INSERT INTO table_2 VALUES ('r1', 'Xpk1', 0);
COMMIT;
Checking for constraints on the table table_2 in the public schema detects a foreign key
violation:
=> SELECT ANALYZE_CONSTRAINTS('public.table_2');
Administrator's Guide
About Constraints
Now add a duplicate value into the unique key and commit the changes:
INSERT INTO table_2 VALUES ('r2', 'Xpk1', 1);
INSERT INTO table_2 VALUES ('r1', 'Xpk1', 1);
COMMIT;
Checking for constraint violations on table table_2 detects the duplicate unique key error:
SELECT ANALYZE_CONSTRAINTS('table_2');
-[RECORD 1 ]---+---------------Schema Name
| public
Table Name
| table_2
Column Names
| z_FK
Constraint Name | C_FOREIGN
Constraint Type | FOREIGN
Column Values
| ('0')
-[RECORD 2 ]---+---------------Schema Name
| public
Table Name
| table_2
Column Names
| x, y_PK
Constraint Name | table_2_multiuk
Constraint Type | PRIMARY
Column Values
| ('r1', 'Xpk1')
Create a table with multicolumn foreign key and create the superprojections:
CREATE TABLE table_3(
z_fk1 VARCHAR(3),
z_fk2 VARCHAR(4));
ALTER TABLE table_3
ADD CONSTRAINT table_3_multifk FOREIGN KEY (z_fk1, z_fk2)
REFERENCES table_2(x, y_PK);
Insert a foreign key that matches a foreign key in table table_2 and commit the changes:
INSERT INTO table_3 VALUES ('r1', 'Xpk1');
COMMIT;
Administrator's Guide
About Constraints
(No rows)
Add a value that does not match and commit the change:
INSERT INTO table_3 VALUES ('r1', 'NONE');
COMMIT;
Administrator's Guide
About Constraints
Column Names
| b_UK
Constraint Name | C_UNIQUE
Constraint Type | UNIQUE
Column Values
| ('1')
-[RECORD 6 ]---+---------------Schema Name
| public
Table Name
| table_1
Column Names
| c_PK
Constraint Name | C_PRIMARY
Constraint Type | PRIMARY
Column Values
| ('1')
-[RECORD 7 ]---+---------------Schema Name
| public
Table Name
| target
Column Names
| a
Constraint Name | C_PRIMARY
Constraint Type | PRIMARY
Column Values
| ('1')
(5 rows)
The next two statements create a table named fact and the pre-join projection that joins fact to
dim.
CREATE TABLE fact(fk INTEGER REFERENCES dim(pk));
Administrator's Guide
About Constraints
CREATE PROJECTION prejoin_p (fk, pk, x) AS SELECT * FROM fact, dim WHERE pk=fk ORDER BY
x;
The following statements load values into table dim. The last statement inserts a duplicate primary
key value of 1:
INSERT INTO dim values (1,1);
INSERT INTO dim values (2,2);
INSERT INTO dim values (1,2); --Constraint violation
COMMIT;
Table dim now contains duplicate primary key values, but you cannot delete the violating row
because of the presence of the pre-join projection. Any attempt to delete the record results in the
following error message:
ROLLBACK: Duplicate primary key detected in FK-PK join Hash-Join (x dim_p), value 1
In order to remove the constraint violation (pk=1), use the following sequence of commands, which
puts the database back into the state just before the duplicate primary key was added.
To remove the violation:
1. Save the original dim rows that match the duplicated primary key:
CREATE TEMP TABLE dim_temp(pk integer, x integer);
INSERT INTO dim_temp SELECT * FROM dim WHERE pk=1 AND x=1; -- original dim row
Administrator's Guide
About Constraints
SELECT REENABLE_DUPLICATE_KEY_ERROR();
The difference between this message and the rollback message in the previous example is that
a fact row contains a foreign key that matches the duplicated primary key, which has been
inserted. A row with values from the fact and dimension table is now in the pre-join projection.
In order for the DELETE statement (Step 3 in the following example) to complete successfully,
extra predicates are required to identify the original dimension table values (the values that are
in the pre-join).
This example is nearly identical to the previous example, except that an additional INSERT
statement joins the fact table to the dimension table by a primary key value of 1:
INSERT INTO
INSERT INTO
INSERT INTO
primary key
INSERT INTO
COMMIT;
dim values
dim values
fact values
value=1
dim values
(1,1);
(2,2);
(1);
TEMP
TEMP
INTO
INTO
Administrator's Guide
About Constraints
3. Remove the duplicate primary keys. These steps also implicitly remove all fact rows with the
matching foreign key.
4. Remove the original row that contains duplicate values:
DELETE FROM dim WHERE pk=1 AND x=1;
Note: The extra predicate (x=1) specifies removal of the original (1,1) row, rather than the
newly inserted (1,2) values that caused the violation.
7. Reinsert the original values back into the fact and dimension table:
INSERT INTO dim SELECT * from dim_temp;
INSERT INTO fact SELECT * from fact_temp;
COMMIT;
Administrator's Guide
Using Table Partitions
Defining Partitions
Displaying partition metadata with the PARTITIONS system table, to display one row per
partition key, per ROS container
HP Vertica provides the following functions that let you manage your partitions and obtain additional
information about them. See the Partition Management Functions in the SQL Reference Manual.
See Also
l
Administrator's Guide
Using Table Partitions
Segmentation defines how data is spread among cluster nodes for fast data purges and query
performance. Table partitioning segregates data on each node.The goal is to distribute data
evenly across multiple database nodes so that all nodes can participate in query execution.
Partitioning specifies how data is organized within individual nodes for distributed computing.
Partitioning attempts to introduce hot spots within the node, providing a convenient way to drop
data and reclaim the disk space.
Note: Use the CREATE PROJECTION statement to definite segmentation. Use the
CREATE TABLE statement to define the paritioning. Logically, the PARTITIONBY clause is
applied after the segmentation clause. See the SQL Reference Manual for details.
To further illustrate the differences, partitioning data by year makes sense if you intend to retain and
drop data at the granularity of a year. On the other hand, segmenting the data by year would be
inefficient, because the node holding data for the current year would likely answer far more queries
than the other nodes.
The following diagram illustrates the flow of segmentation and partitioning on a four-node database
cluster:
1. Example table data
2. Data segmented by HASH(order_id)
3. Data segmented by hash across four nodes
4. Data partitioned by year on a single node
While partitioning occurs on all four nodes, the illustration shows partitioned data on one node for
simplicity.
Administrator's Guide
Using Table Partitions
See Also
l
Projection Segmentation
CREATE PROJECTION
CREATE TABLE
Defining Partitions
The first step in defining data partitions is to establish the relationship between the data and
partitions. To illustrate, consider the following table called trade, which contains unpartitioned data
for the trade date (tdate), ticker symbol (tsymbol), and time (ttime).
Table 1: Unpartitioned data
tdate
| tsymbol | ttime
------------+---------+---------2008-01-02 | AAA
| 13:00:00
2009-02-04 | BBB
| 14:30:00
Administrator's Guide
Using Table Partitions
2010-09-18 | AAA
2009-05-06 | AAA
2008-12-22 | BBB
(5 rows)
| 09:55:00
| 11:14:30
| 15:30:00
If you want to discard data once a year, a logical choice is to partition the table by year. The partition
expression PARTITION BY EXTRACT(year FROM tdate)creates the partitions shown in Table 2:
Table 2: Data partitioned by year
2008
2009
2010
tdate
tsymbol
ttime
---------+---------+--------01/02/08 | AAA
| 13:00:00
12/22/08 | BBB
| 15:30:00
tdate
tsymbol
ttime
---------+---------+--------02/04/09 | BBB
| 14:30:00
05/06/09 | AAA
| 11:14:30
tdate
tsymbol
ttime
---------+---------+--------09/18/10 | AAA
| 09:55:00
Unlike some databases, which require you to explicitly define partition boundaries in the CREATE
TABLE statement, HP Vertica selects a partition for each row based on the result of a partitioning
expression provided in the CREATE TABLE statement. Partitions do not have explicit names
associated with them. Internally, HP Vertica creates a partition for each distinct value in the
PARTITION BY expression.
After you specify a partition expression, HP Vertica processes the data by applying the partition
expression to each row and then assigning partitions.
The following syntax generates the partitions for this example, with the results shown in Table 3. It
creates a table called trade, partitioned by year. For additional information, see CREATE TABLE
in the SQL Reference Manual.
CREATE TABLE trade (
tdate DATE NOT NULL,
tsymbol VARCHAR(8) NOT NULL,
ttime TIME)
PARTITION BY EXTRACT (year FROM tdate);
CREATE PROJECTION trade_p (tdate, tsymbol, ttime) AS
SELECT * FROM trade
ORDER BY tdate, tsymbol, ttime UNSEGMENTED ALL NODES;
INSERT INTO trade VALUES ('01/02/08' , 'AAA' , '13:00:00');
INSERT INTO trade VALUES ('02/04/09' , 'BBB' , '14:30:00');
INSERT INTO trade VALUES ('09/18/10' , 'AAA' , '09:55:00');
INSERT INTO trade VALUES ('05/06/09' , 'AAA' , '11:14:30');
INSERT INTO trade VALUES ('12/22/08' , 'BBB' , '15:30:00');
Administrator's Guide
Using Table Partitions
You can use the following partition expression to partition the table using the year and month:
PARTITION BY EXTRACT(year FROM tdate)*100 + EXTRACT(month FROM tdate)
The partitioning expression can reference one or more columns from the table.
The partitioning expression cannot evaluate to NULL for any row, so do not include columns that
allow a NULL value in the CREATE TABLE..PARTITION BY expression.
Any SQL functions in the partitioning expression must be immutable, meaning that they return
the exact same value regardless of when they are invoked, and independently of session or
environment settings, such as LOCALE. For example, you cannot use the TO_CHAR function
in a partition expression, because it depends on locale settings, or the RANDOM function, since
it produces different values at each invocation.
All projections anchored on a table must include all columns referenced in the PARTITION BY
expression; this allows the partition to be calculated.
Administrator's Guide
Using Table Partitions
While HP Vertica supports a maximum of 1024 partitions, few, if any, organizations will need to
approach that maximum. Fewer partitions are likely to meet your business needs, while also
ensuring maximum performance. Many customers, for example, partition their data by month,
bringing their partition count to 12. HP Vertica recommends you keep the number of partitions
between 10 and 20 to achieve excellent performance.
Do not apply partitioning to tables used as dimension tables in pre-join projections. You can
apply partitioning to tables used as large single (fact) tables in pre-join projections.
For maximum performance, do not partition projections on LONG VARBINARY and LONG
VARCHAR columns.
Dropping Partitions
Use the DROP_PARTITION function to drop a partition. Normally, this is a fast operation that
discards all ROS containers that contain data for the partition.
Occasionally, a ROS container contains rows that belong to more than one partition. For example,
this can happen after a MERGE_PARTITIONS operation. In this case, HP Vertica performs a split
operation to avoid discarding too much data. HP Vertica tries to keep data from different partitions
segregated into different ROS containers, but there are a small number of exceptions. For instance,
the following operations can result in a ROS container with mixed partitions:
l
MERGE_PARTITIONS, which merges ROS containers that have data belonging to partitions in
a specified partition key range
Refresh and recovery operations can generate ROS containers with mixed partitions under
some conditions. See Auto Partitioning.
The number of partitions that contain data is restricted by the number of ROS containers that can
comfortably exist in the system.
In general, if a ROS container has data that belongs to n+1 partitions and you want to drop a
specific partition, the DROP_PARTITION operation:
Administrator's Guide
Using Table Partitions
one container holds the data that belongs to the partition that is to be dropped
Examples
Using the example schema in Defining Partitions, the following command explicitly drops the 2009
partition key from table trade:
SELECT DROP_PARTITION('trade', 2009);
DROP_PARTITION
------------------Partition dropped
(1 row)
The following example creates a table called dates and partitions the table by year:
CREATE TABLE dates (year INTEGER NOT NULL,
month VARCHAR(8) NOT NULL)
PARTITION BY year * 12 + month;
Administrator's Guide
Using Table Partitions
The following statement drops the partition using a constant for Oct 2010 (2010*12 + 10 = 24130):
SELECT DROP_PARTITION('dates', '24130');
DROP_PARTITION
------------------Partition dropped
(1 row)
See Also
l
DROP_PARTITION
Swapping Partitions
To swap partitions between two tables, those tables must have identical definitions, including the
same:
l
Column definitions
Segmentation
Partitioning expression
Number of projections
Swapping partitions is useful if you regularly load new or updated data into a table.
Suppose you have a table of revenue that is partitioned by date. On a regular basis, you need to
move the data from a staging table to the main table. If you have new or modified data for a past
date, the previously created partitions require updating. You must delete the old partitions and move
the data from the corresponding partitions of the staging table into a new partition.
Administrator's Guide
Using Table Partitions
Temporary tables
Virtual tables
System tables
External tables
For a step-by-step tutorial on swapping partitions, see Tutorial for Swapping Partitions.
Data is automatically split into partitions during load / refresh / recovery operations.
Each ROS container contains data for a single partition, though there can be multiple ROS
containers for a single partition.
Partition Pruning
When a query predicate includes one more more columns in the partitioning clause, queries look
only at relevant ROS containers. See Partition Elimination for details.
Administrator's Guide
Using Table Partitions
Managing Partitions
HP Vertica provides various options to let you manage and monitor the partitions you create.
PARTITION_TABLE() function
The PARTITION_TABLE() function physically separates partitions into separate containers. Only
ROS containers with more than one distinct value participate in the split.
The following example creates a simple table called states and partitions the data by state.
=> CREATE TABLE states (year INTEGER NOT NULL,
state VARCHAR NOT NULL)
PARTITION BY state;
=> CREATE PROJECTION states_p (state, year) AS
SELECT * FROM states
ORDER BY state, year UNSEGMENTED ALL NODES;
Administrator's Guide
Using Table Partitions
Notes
There are just a few more things worth mentioning to help you manage your partitions:
l
To prevent too many ROS containers, be aware that delete operations must open all the
containers; thus, ideally create fewer than 20 partitions and avoid creating more than 50.
Your partition-clause must calculate a single non-null value for each row. You can reference
multiple columns, but each row must return a single value.
You can specify leaf expressions, functions, and operators in the partition clause expression.
All leaf expressions in the partition clause must be either constants or columns of the table.
Aggregate functions and queries are not permitted in the partition-clause expression.
Administrator's Guide
Using Table Partitions
Partitioning or re-partitioning tables requires USAGE privilege on the schema that contains the
table.
Monitoring Reorganization
When you use the ALTER TABLE .. REORGANIZE, the operation reorganizes the data in the
background.
You can monitor details of the reorganization process by polling the following system tables:
l
V_MONITOR.PARTITIONS displays NULLS in the partition_key column for any ROS's that
have not been reorganized.
Administrator's Guide
Using Table Partitions
Auto Partitioning
HP Vertica attempts to keep data from each partition stored separately. Auto partitioning occurs
when data is written to disk, such as during COPY DIRECT or moveout operations.
Separate storage provides two benefits: Partitions can be dropped quickly, and partition elimination
can omit storage that does not need to need not to participate in a query plan.
Note: If you use INSERT...SELECT in a partitioned table, HP Vertica sorts the data before
writing it to disk, even if the source of the SELECT has the same sort order as the destination.
Examples
The examples that follow use this simple schema. First, create a table named t1 and partition the
data on the c1 column:
CREATE TABLE
c1 INT NOT
c2 INT NOT
SEGMENTED BY
PARTITION BY
t1 (
NULL,
NULL)
c1 ALL NODES
c2;
INTO
INTO
INTO
INTO
t1
t1
t1
t1
VALUES(10,15);
VALUES(20,25);
VALUES(30,35);
VALUES(40,45);
Administrator's Guide
Using Table Partitions
SELECT DO_TM_TASK('moveout','t1');
do_tm_task
-------------------------------moveout for projection 't1_p1'
moveout for projection 't1_p'
(1 row)
Query the PARTITIONS system table. Notice that the four partition keys reside on two nodes, each
in its own ROS container (see the ros_id column). The PARTITION BY clause was used on
column c2, so HP Vertica auto partitioned the input values during the COPY operation:
SELECT partition_key, projection_name, ros_id, ros_size_bytes,
ros_row_count, node_name FROM PARTITIONS
WHERE projection_name like 't1_p1';
partition_key | projection_name |
ros_id
| ros_size_bytes | ros_row_count |
node_name
---------------+-----------------+-------------------+----------------+---------------+-----------------15
| t1_p1
| 49539595901154617 |
78 |
1 |
node0002
25
| t1_p1
| 54043195528525081 |
78 |
1 |
node0003
35
| t1_p1
| 54043195528525069 |
78 |
1 |
node0003
45
| t1_p1
| 49539595901154605 |
79 |
1 |
node0002
(4 rows)
HP Vertica does not auto partition when you refresh with the same sort order. If you create a new
projection, HP Vertica returns a message telling you to refresh the projections; for example:
CREATE PROJECTION t1_p2 AS SELECT * FROM t1 SEGMENTED BY HASH(c1) ALL NODES OFFSET 2;
WARNING: Projection <public.t1_p2> is not available for query processing. Execute the
select
start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes
must be up before starting a refresh.
Query the PARTITIONS system table again. The partition keys now reside in two ROS containers,
instead of four, which you can tell by looking at the values in the ros_id column. The ros_row_
count column holds the number of rows in the ROS container:
Administrator's Guide
Using Table Partitions
The following command more specifically queries ROS information for the partitioned tables. In this
example, the query counts two ROS containers each on two different nodes for projection t1_p2:
SELECT ros_id, node_name, COUNT(*) FROM PARTITIONS
WHERE projection_name LIKE 't1_p2'
GROUP BY ros_id, node_name;
ros_id
| node_name | COUNT
-------------------+-----------+------54043195528525121 | node0003 |
2
58546795155895541 | node0004 |
2
(2 rows)
This command returns a result of four ROS containers on two different nodes for projection t1_p1:
SELECT ros_id,node_name, COUNT(*) FROM PARTITIONS
WHERE projection_name LIKE 't1_p1'
GROUP BY ros_id, node_name;
ros_id
| node_name | COUNT
-------------------+-----------+------49539595901154605 | node0002 |
1
49539595901154617 | node0002 |
1
54043195528525069 | node0003 |
1
54043195528525081 | node0003 |
1
(4 rows)
See Also
l
DO_TM_TASK
PARTITIONS
START_REFRESH
Administrator's Guide
Using Table Partitions
Eliminating Partitions
If the ROS containers of partitioned tables are not needed, HP Vertica can eliminate the containers
from being processed during query execution. To eliminate ROS containers, HP Vertica compares
query predicates to partition-related metadata.
Each ROS partition expression column maintains the minimum and maximum values of data stored
in that ROS, and HP Vertica uses those min/max values to potentially eliminate ROS containers
from query planning. Partitions that cannot contain matching values are not scanned. For example,
if a ROS does not contain data that satisfies a given query predicate, the optimizer eliminates
(prunes) that ROS from the query plan. After non-participating ROS containers have been
eliminated, queries that use partitioned tables run more quickly.
Note: Partition pruning occurs at query run time and requires a query predicate on the
partitioning column.
Assume a table that is partitioned by year (2007, 2008, 2009) into three ROS containers, one for
each year. Given the following series of commands, the two ROS containers that contain data for
2007 and 2008 fall outside the boundaries of the requested year (2009) and get eliminated.
=> CREATE TABLE ... PARTITION BY EXTRACT(year FROM date);
=> SELECT ... WHERE date = '12-2-2009';
On any database that has been upgraded from version 3.5, or earlier, ROS containers are ineligible
for partition elimination because they do not contain the minimum/maximum partition key values
required. These ROS containers need to be recreated or merged by the Tuple Mover.
Administrator's Guide
Using Table Partitions
2. Identify which partitions need to be merged to get the ROS minimum/maximum values by
running the following command:
=> SELECT DISTINCT table_schema, projection_name, partition_key
FROM partitions p LEFT OUTER JOIN vs_ros_min_max_values v
ON p.ros_id = v.delid
WHERE v.min_value IS null;
3. Insert a record into each partition that has ineligible ROS containers and commit.
4. Delete each inserted record and commit again.
At this point, the Tuple Mover automatically merges ROS containers from past partitions.
Administrator's Guide
Using Table Partitions
Examples
Assume a table that is partitioned by time and will use queries that restrict data on time.
CREATE TABLE time (
tdate DATE NOT NULL,
tnum INTEGER)
PARTITION BY EXTRACT(year FROM tdate);
CREATE PROJECTION time_p (tdate, tnum) AS
SELECT * FROM time
ORDER BY tdate, tnum UNSEGMENTED ALL NODES;
INSERT
INSERT
INSERT
INSERT
INTO
INTO
INTO
INTO
time
time
time
time
VALUES
VALUES
VALUES
VALUES
('03/15/04'
('03/15/05'
('03/15/06'
('03/15/06'
,
,
,
,
1);
2);
3);
4);
The data inserted in the previous series of commands would be loaded into three ROS containers,
one per year, since that is how the data is partitioned:
SELECT * FROM time ORDER BY tnum;
tdate
| tnum
------------+-----2004-03-15 |
1 --ROS1 (min 03/01/04,
2005-03-15 |
2 --ROS2 (min 03/15/05,
2006-03-15 |
3 --ROS3 (min 03/15/06,
2006-03-15 |
4 --ROS3 (min 03/15/06,
(4 rows)
max
max
max
max
03/15/04)
03/15/05)
03/15/06)
03/15/06)
In the this query, HP Vertica can eliminate ROS2 because it is only looking for year 2004:
=> SELECT COUNT(*) FROM time WHERE tdate = '05/07/2004';
In the next query, HP Vertica can eliminate both ROS1 and ROS3:
Administrator's Guide
Using Table Partitions
The following query has an additional predicate on the tnum column for which no
minimum/maximum values are maintained. In addition, the use of logical operator OR is not
supported, so no ROS elimination occurs:
=> SELECT COUNT(*) FROM time WHERE tdate = '05/07/2004' OR tnum = 7;
Moving Partitions
You can move partitions from one table to another using the MOVE_PARTITIONS_TO_TABLE
function. Use this function when you create offline archives of older partitions. By moving partitions
from one table to an intermediate table, you can then create a backup of the new table, and drop the
partition. If you need the historical data later, you can restore the archived partitions, described in
Restoring Archived Partitions.
If the target table does not exist, the MOVE_PARTITIONS_TO_TABLE function creates a table
definition using the CREATE TABLE statement with its LIKE clause. You create a table with the
LIKE clause using a DDL operation. HP Vertica does not copy any data from the source table, and
the new table is not connected to its source in any way. The CREATE TABLE statement with the
LIKE clause does not copy constraints, automatic values (such as a sequences and identity
values), or a default values. Corresponding columns in the new table have the same type as the
source table, but the columns do not have constraints or automatic values.
Archiving Steps
To archive partitions:
1. Prepare and move the partitions with the MOVE_PARTITIONS_TO_TABLE function.
2. Create an object-level snapshot of the intermediate table.
3. Drop the intermediate table.
The next sections describe the archiving steps.
Administrator's Guide
Using Table Partitions
Check that the name you plan to use does not conflict with an existing table name.
Use a name that represents the partition values you are moving.
When you have created a separate schema for the intermediate table, call the MOVE_
PARTITIONS_TO_TABLE function.
If you call MOVE_PARTITIONS_TO_TABLE and the target table does not exist, the function
creates the table automatically. In the following example, the target table partn_backup.tradfes_
200801 does not exist. MOVE_PARTITIONS_TO_TABLEcreates the table and moves the
partition. In addition, HP Vertica copies all the constraints associated with the source table except
any foreign key constraints.
=> SELECTMOVE_PARTITIONS_TO_TABLE (
'prod_trades',
'200801',
'200801',
'partn_backup.trades_200801');
MOVE_PARTITIONS_TO_TABLE
------------------------------------------------1 distinct partition values moved at epoch 15.
(1 row)
Administrator's Guide
Using Table Partitions
See Also
l
CREATE TABLE
Swapping Partitions
SWAP_PARTITIONS_BETWEEN_TABLES
Administrator's Guide
Using Table Partitions
2. Move the restored partitions from the intermediate table to the original table.
3. Drop the intermediate table.
See Also
l
Administrator's Guide
Using Table Partitions
Administrator's Guide
Bulk Loading Data
The COPY statement loads data from a file stored on the host or client (or in a data stream) into a
database table. You can pass the COPY statement many different parameters to define various
options such as:
l
HP Vertica's hybrid storage model provides a great deal of flexibility for loading and managing data.
Administrator's Guide
Bulk Loading Data
See the remaining sections here for other options, and the COPY statement in the SQL Reference
Manual for syntax details.
Administrator's Guide
Bulk Loading Data
The file command could indicate ASCII TEXT even though the file contains multibyte characters.
To check for multibyte characters in an ASCII file, use the wc command. For example:
$wc Date_Dimension.tbl
1828
5484 221822 Date_Dimension.tbl
See the man pages for file and iconv for more information.
If all of the strings are in UTF-8 format, the query should not return any rows.
Administrator's Guide
Bulk Loading Data
Tip: HP Vertica supports multiple schema types. If you have a starschema, load the smaller
tables before you load the largest tables.
Only a superuser can use the COPY statement to bulk load data. Two exceptions to the superuser
requirement are to:
1. Run COPY to load from a stream on the host (such as STDIN) rather than a file (see Streaming
Data via JDBC).
2. Use the COPY statement with the FROM LOCAL option.
A non-superuser can also perform a batch load with a JDBC prepared statement, which invokes
COPY to load data as a background task.
Administrator's Guide
Bulk Loading Data
If only a few rows contain |, you can eliminate them from the load file using a WHERE clause and
load them separately using a different delimiter.
Tip: : For loading data from an Oracle database, use a WHERE clause to avoid problem rows
in the main load file, and the negated WHERE clause with REGEX_REPLACE for problem
rows.
Deliver chunks of data to the different HP Vertica nodes by connecting the transport disk or by
writing files from network copy.
Even without a filesystem on the disk, plug-and-play support still works on Linux to provide a device
node for the disk. To find out the assigned device, plug in the disk and enter:
$dmesg | tail -40
SATA disks are usually internal, but can be external, or unmounted safely if they are internal.
Administrator's Guide
Bulk Loading Data
2. Create another vsql variable that uses a path relative to the Linux current directory variable for
Administrator's Guide
Bulk Loading Data
Note: COPY FROM LOCAL does not require an absolute path for data files. You can use
paths that are relative to the client's directory system.
Administrator's Guide
Bulk Loading Data
To use one of the several flex table parsers, use the parser parameter, followed by the parser of
choice.
COPY has many options, which you can combine to make importing data flexible. For detailed
syntax of the various options see the SQL Reference Manual. For example:
For this option...
compressed forms.
Options
Insert data into the WOS (memory) or directly into the ROS (disk).
You can use a comma-separated list to load multiple files of the same compression type. COPY
LOCAL then concatenates the files into a single file, so you cannot combine files with different
compression types in the list. When listing multiple files, be sure to specify the type of every input
file, such as BZIP, as shown:
=>COPY simple_table FROM LOCAL 'input_file.bz' BZIP, 'input_file.bz' BZIP;
You can load data from a local client from STDIN, as follows:
Administrator's Guide
Bulk Loading Data
Administrator's Guide
Bulk Loading Data
Date/time
Formatting Functions
String
Null-handling
System information
COPY expressions cannot use SQL meta functions (HP Vertica-specific), analytic functions,
aggregate functions, or computed columns.
For computed columns, all parsed columns in the expression must be listed in the COPY statement.
Do not specify FORMAT or RAW in the source data for a computed column.
Expressions used in a COPY statement can contain only constants. The return data type of the
expression must be coercible to that of the target column. Parsed column parameters are also
coerced to match the expression.
Administrator's Guide
Bulk Loading Data
Transformation Example
Following is a small transformation example.
1. Create a table and corresponding projection.
CREATE TABLE t (
year VARCHAR(10),
month VARCHAR(10),
day VARCHAR(10),
k timestamp
);
CREATE PROJECTION tp (
year,
month,
day,
k)
AS SELECT * from t;
2. Use COPY to copy the table, computing values for the year, month, and day columns in the
target database, based on the timestamp columns in the source table.
3. Load the parsed column, timestamp, from the source data to the target database.
COPY t(year AS TO_CHAR(k, 'YYYY'),
month AS TO_CHAR(k, 'Month'),
day AS TO_CHAR(k, 'DD'),
k FORMAT 'YYYY-MM-DD') FROM STDIN NO COMMIT;
2009-06-17
1979-06-30
2007-11-26
\.
Administrator's Guide
Bulk Loading Data
The next example illustrates how to use the year, month, and day columns from the source input to
derive and load the value for the TIMESTAMP column in the target database.
1. Create a table and corresponding projection:
=> CREATE TABLE t (k TIMESTAMP);=> CREATE PROJECTION tp (k) AS SELECT * FROM t;
2. Use COPY with the FILLER keyword to skip the year, month, and day columns from the
source file.
=> COPY t(year FILLER VARCHAR(10),
month FILLER VARCHAR(10),
day FILLER VARCHAR(10),
k AS TO_DATE(YEAR || MONTH || DAY, 'YYYYMMDD') )
FROM STDIN NO COMMIT;
>> 2009|06|17
>> 1979|06|30
>> 2007|11|26
>> \.
See also Using Sequences for how to generate an auto-incrementing value for columns.
See the COPY statement in the SQL Reference Manual for further information.
Administrator's Guide
Bulk Loading Data
Each of the FROM keywords lets you optionally specify the format of the load file as
UNCOMPRESSED, BZIP, GZIP, or LZO.
Note: When using COPY in conjunction with a CREATE EXTERNAL TABLE statement, you
cannot use the COPY FROM STDIN or LOCAL options.
The storage location must have been created with the USER option (see CREATE LOCATION)
The user must already have been granted READ access to the storage location where the file(s)
exist, as described in GRANT (Storage Location)
Further, if a non-superuser invokes COPY from a storage location to which that user has privileges,
HP Vertica also checks any symbolic links (symlinks) the user has to ensure no symlink can
access an area to which the user has not been granted privileges.
Administrator's Guide
Bulk Loading Data
------------172094
(1 row)
HP Vertica supports the following options to the lzop command used to compress the file:
l
Using a wildcard with the ON ANY NODE clause expands the file list on the initiator node. This
command then distributes the individual files among all nodes, so that the COPY workload is
evenly distributed across the entire cluster.
Administrator's Guide
Bulk Loading Data
For specific information about saving rejected data and exceptions files when using COPY from
LOCAL, see Capturing Load Rejections and Exceptions.
AUTO
Loads data into WOS. Use this default COPY load method for smaller bulk loads.
DIRECT
Loads data directly into ROS containers. Use the DIRECT load method for large bulk
loads (100MB or more).
TRICKLE
Loads data only into WOS. Use for frequent incremental loads, after the initial bulk
load is complete.
Note: COPY ignores any load method you specify as part of creating an external table.
Note: A large initial bulk load can temporarily affect query performance while HP Vertica
organizes the data.
Administrator's Guide
Bulk Loading Data
Using a single transaction for multiple COPY statements also allows HP Vertica to load the data
more efficiently since it can combine the larger amounts of data from multiple COPY statements
into fewer ROS containers.
HP recommends that you COMMIT or ROLLBACK the current transaction before you use COPY.
You can combine NO COMMIT with most other existing COPY options, but not the
REJECTEDDATAASTABLE option. The standard transaction semantics apply. If a transaction is in
progress that was initiated by a statement other than COPY (such as INSERT), using COPY with NO
COMMIT adds rows to the existing transaction, rather than starting a new one. The previous
statements are NOT committed.
Note: NO COMMIT is ignored when COPY is part of the CREATE EXTERNAL TABLE FROM
COPY statement.
Administrator's Guide
Bulk Loading Data
NATIVE
NATIVE VARCHAR
FIXEDWIDTH
ORC
Note: You do not specify the DELIMITER parser directly; absence of a specific parser indicates
Administrator's Guide
Bulk Loading Data
the default.
Using a different parser for your data can improve load performance. If delimited input data includes
binary data types, COPY translates the data on input. See Using Load Scripts and Loading Binary
(Native) Data for examples. You can also load binary data, but only if it adheres to the COPY format
requirements, described in Creating Native Binary Format Files.
You cannot mix raw data types that require different parsers (such as NATIVE and FIXEDWIDTH)
in a single bulk load COPY statement. To check data formats before (or after) loading, see
Checking Data Format Before or After Loading.
FCEFPARSER
FDELIMITEDPARSER
FDELIMITEDPAIRPARSER
FJSONPARSER
FREGEXPARSER
Loading data with the flex parsers makes loading data flexible. For instance, you can load
JSONdata into a columnar table in one load with the FJSONPARSER, and delimited data into the
same table in another with the default COPYparser. The Flex Tables Guide describes this use
case and presents an example.
Administrator's Guide
Bulk Loading Data
Metadata Option
As a Column or Expression
Option
As a
COLUMN
OPTION
As a FROM
Level Option
DELIMITER
ENCLOSED BY
ESCAPE AS
NULL
TRIM
RECORD
TERMINATOR
SKIP
SKIP BYTES
Y (Fixed-width
only)
TRAILING
NULLCOLS
The following precedence rules apply to all data loads:
l
COPY uses the statement-level parameter if you do not specify a column-level parameter.
COPY uses the default metadata values for the DELIMITER, ENCLOSED BY, ESCAPE AS,
and NULL options if you do not specify them at either the statement- or column-level.
When you specify any metadata options, COPY uses the parser to produce the best results and
stores the raw data and its corresponding metadata in the following formats:
UTF-8
DELIMITER
Binary
Binary
NATIVE
UTF-8
Binary
NATIVE VARCHAR
UTF-8
UTF-8
FIXEDWIDTH
Administrator's Guide
Bulk Loading Data
See Also
l
COPY
Record terminator
EOF designator
Note: The FIXEDWIDTH parser always requires exactly a record terminator. No other
permutations work.
For example, given a three-column table, the following input rows for a COPY statement using a
comma (,) delimiter are each valid:
1,1,11,1,1,
1,1,
1,1,,
The following examples illustrate how COPY can interpret different last column end of data row
values.
2. COPY the two_col table using a comma (,) delimiter, and enter values for only one column (as
a single, multi-line entry):
Administrator's Guide
Bulk Loading Data
Here, COPY expects two values for each column, but gets only one. Each input value is followed
by a delimiter (,), and an implicit record terminator (a newline character, \n). You supply a record
terminator with the ENTER or RETURN key. This character is not represented on the screen.
In this case, the delimiter (,) and record terminator (\n) are handled independently. COPY interprets
the delimiter (,) to indicate the end of one value, and the record terminator (\n) to specify the end of
the column row. Since no value follows the delimiter, COPY supplies an empty string before the
record terminator. By default, the empty string signifies a NULL, which is a valid column value.
Administrator's Guide
Bulk Loading Data
In this example, COPY expects values for only one column, because of the column (a) directive.
As such, COPY interprets the delimiter and record terminator together as a single, valid, last
column end of row definition. Before parsing incoming data, COPY populates column b with its
default value, because the table definition has two columns and the COPY statement supplies only
one. This example populates the second column with its default column list value, while the
previous example used the supplied input data.
ENCLOSED BY
ESCAPE AS
NULL
DELIMITER
Backslash (\)
Various
Administrator's Guide
Bulk Loading Data
To use a special character as a literal, prefix it with an escape character. For example, to include a
literal backslash (\) in the loaded data (such as when including a file path), use two backslashes
(\\). COPY removes the escape character from the input when it loads escaped characters.
TABLE t(
pk INT,
VARCHAR(10),
VARCHAR(10),
VARCHAR(10),
TIMESTAMP);
2. Use the COLUMN OPTION parameter to change the col1 default delimiter to a tilde (~).
=> COPY t COLUMN OPTION(col1 DELIMITER '~') FROM STDIN NO COMMIT;
Administrator's Guide
Bulk Loading Data
>> 1|ee~gg|yy|1999-12-12
>> \.
=> SELECT * FROM t;
pk | col1 | col2 | col3 |
col4
----+------+------+------+--------------------1 | ee
| gg
| yy
| 1999-12-12 00:00:00
(1 row)
To input an empty or literal string, use quotes (ENCLOSED BY); for example:
NULL ''NULL 'literal'
A NULL is case-insensitive and must be the only value between the data field delimiters. For
example, if the null string is NULL and the delimiter is the default vertical bar (|):
|NULL| indicates a null value.
| NULL | does not indicate a null value.
When you use the COPY command in a script, you must substitute a double-backslash for each null
string that includes a backslash. For example, the scripts used to load the example database
contain:
COPY ... NULL E'\\n' ...
Administrator's Guide
Bulk Loading Data
|
4
6
|
|
|
1| 2
| 5
|
| 3
2 |
as follows:
(null, null, 1)(null, 2, 3)
(4, null, 5)
(6, null, null)
Administrator's Guide
Bulk Loading Data
5. Query table z again to see the results. Using TRAILING NULLCOLS, the COPY statement
correctly handled the third row of column c, which had no value:
=> SELECT * FROM z;
a | b | c
---+---+--1 | 2 | 3
4 | 5 | 6
7 | 8 |
(3 rows)
Administrator's Guide
Bulk Loading Data
(1 row)
3. Use COPY with TRAILING NULLCOLS on table n to see the COPY error due to the column
constraint:
=> COPY n FROM STDIN trailing nullcols abort on error;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 4 | 5 | 6
>> 7 | 8
>> 9
>> \.
ERROR: COPY: Input record 3 has been rejected (Cannot set trailing column to NULL as
column 2 (b) is NOT NULL)
4. Query the table to see that the COPY statement values were rejected:
=> SELECT * FROM n;
a | b | c
---+---+--1 | 2 | 3
(1 row)
You can set the escape character to be any ASCII value value in the range E'\001' to E'\177'
inclusive.
Administrator's Guide
Bulk Loading Data
(") is the most commonly used quotation character. For instance, the following parameter specifies
that input data to the COPY statement is enclosed within double quotes:
ENCLOSED BY '"'
With the following input (using the default DELIMITER (|) character), specifying:
"vertica | value"
Results in:
l
Notice the double quotes (") before vertica and after value.
Using the following sample input data as follows, columns are distributed as shown:
"1", "vertica,value", ",", "'"
col1 | col2
| col3 | col4------+---------------+------+----1
| vertica,value | ,
| '
(1 row)
Alternatively, write the above example using any ASCII character of your choosing:
~1~, ~vertica,value~, ~,~, ~'~
If you use a single quote ('), rather than double quotes (") as the ENCLOSED BY character, you must
escape it using extended string syntax, a Unicode literal string (if StandardConformingStrings is
enabled), or by using four single quotes:
ENCLOSED BY E'\''ENCLOSED BY U&'\0027'
ENCLOSED BY ''''
Using any of the definitions means the following input is properly parsed:
'1', 'vertica,value', ',', '\''
See String Literals (Character) for an explanation of the string literal formats you can use to specify
the ENCLOSED BY parameter.
Administrator's Guide
Bulk Loading Data
Use the ESCAPE AS character to embed the ENCLOSED BY delimiter within character string values.
For example, using the default ESCAPE AS character (\) and double quote as the ENCLOSED BY
character, the following input returns "vertica":
"\"vertica\""
To specify the RECORD TERMINATOR as non-printing characters, use either the extended string
syntax or Unicode string literals. The following table lists some common record terminator
characters. See String Literals for an explanation of the literal string formats.
Extended String Syntax Unicode Literal String
Description
ASCII Decimal
E'\b'
U&'\0008'
Backspace
E'\t'
U&'\0009'
Horizontal tab
E'\n'
U&'\000a'
Linefeed
10
Administrator's Guide
Bulk Loading Data
E'\f'
U&'\000c'
Formfeed
12
E'\r'
U&'\000d'
Carriage return
13
E'\\'
U&'\005c'
Backslash
92
If you use the RECORD TERMINATOR option to specify a custom value, be sure the input file matches
the value. Otherwise, you may get inconsistent data loads.
Note: The record terminator cannot be the same as DELIMITER, NULL, ESCAPE,or ENCLOSED
BY.
If using JDBC, HP recommends that you use the following value for the RECORD TERMINATOR:
System.getProperty("line.separator")
Examples
The following examples use a comma (,) as the DELIMITER for readability.
,1,2,3,,1,2,3
1,2,3,
Leading (,1) and trailing (3,) delimiters are ignored. Thus, the rows all have three columns.
123,'\\n',\\n,456
Administrator's Guide
Bulk Loading Data
Using NATIVE VARCHAR does not provide the same efficiency as NATIVE. However, NATIVE
VARCHAR precludes the need to use delimiters or to escape special characters, such as quotes,
which can make working with client applications easier.
Note: NATIVE VARCHAR does not support concatenated compressed files.
Batch data inserts performed through the HP Vertica ODBC and JDBC drivers automatically use
the NATIVE VARCHAR format.
Hexadecimal
Octal
Bitstring
Administrator's Guide
Bulk Loading Data
hex VARBINARY(5),
bitstring VARBINARY(5) );
4. Enter the data to load, ending the statement with a backslash (\) and a period (.) on a separate
line:
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 141142143144145,0x6162636465,0110000101100010011000110110010001100101
>> \.
COPY uses the same default format to load binary data, as used to input binary data. Since the
backslash character ('\') is the default escape character, you must escape octal input values. For
example, enter the byte '\141' as '\\141'.
Note: If you enter an escape character followed by an invalid octal digit or an escape character
being escaped, COPYreturns an error.
On input, COPY translates string data as follows:
l
Administrator's Guide
Bulk Loading Data
You can also use the escape character to represent the (decimal) byte 92 by escaping it twice; for
example, '\\\\'. Note that vsql inputs the escaped backslash as four backslashes. Equivalent
inputs are hex value '0x5c' and octal value '\134' (134 = 1 x 8^2 + 3 x 8^1 + 4 x 8^0 = 92).
You can load a delimiter value if you escape it with a backslash. For example, given delimiter '|',
'\\001\|\\002' is loaded as {1,124,2}, which can also be represented in octal format as
'\\001\\174\\002'.
If you insert a value with more bytes than fit into the target column, COPY returns an error. For
example, if column c1 is VARBINARY(1):
=> INSERT INTO t (c1) values ('ab');
If you implicitly or explicitly cast a value with more bytes than fit the target data type, COPY silently
truncates the data. For example:
=> SELECT 'abcd'::binary(2);
binary
-------ab
(1 row)
Hexadecimal Data
The optional '0x' prefix indicates that a value is hexadecimal, not decimal, although not all
hexadecimal values use A-F; for example, 5396. COPY ignores the 0x prefix when loading the input
data.
If there are an odd number of characters in the hexadecimal value, the first character is treated as
the low nibble of the first (furthest to the left) byte.
Octal Data
Loading octal format data requires that each byte be represented by a three-digit octal code. The
first digit must be in the range [0,3] and the second and third digits must both be in the range [0,7].
If the length of an octal value is not a multiple of three, or if one of the three digits is not in the proper
range, the value is invalid and COPY rejects the row in which the value appears. If you supply an
invalid octal value, COPY returns an error. For example:
SELECT '\\000\\387'::binary(8);
ERROR: invalid input syntax for type binary
Rows that contain binary values with invalid octal representations are also rejected. For example,
COPY rejects '\\008' because '\\ 008' is not a valid octal number.
Administrator's Guide
Bulk Loading Data
BitString Data
Loading bitstring data requires that each character must be zero (0) or one (1), in multiples of eight
characters. If the bitstring value is not a multiple of eight characters, COPY treats the first n
characters as the low bits of the first byte (furthest to the left), where n is the remainder of the
value's length, divided by eight.
Examples
The following example shows VARBINARY HEX_TO_BINARY(VARCHAR) and VARCHAR TO_HEX
(VARBINARY) usage.
1. Create table t and and its projection with binary columns:
=> CREATE TABLE t (c BINARY(1));
=> CREATE PROJECTION t_p (c) AS SELECT c FROM t;
2. Insert minimum and maximum byte values, including an IP address represented as a character
string:
=> INSERT INTO t values(HEX_TO_BINARY('0x00'));
=> INSERT INTO t values(HEX_TO_BINARY('0xFF'));
=> INSERT INTO t values (V6_ATON('2001:DB8::8:800:200C:417A'));
See Also
l
COPY
Administrator's Guide
Bulk Loading Data
Formatting Functions
ASCII
DELIMITER
ENCLOSED BY
ESCAPE AS
TRAILING NULLCOLS
Statement levelNULL must be defined as a single-character. The default (or custom) NULL
character is repeated for the entire width of the column.
Column LevelNULL must be defined as a string whose length matches the column width.
Administrator's Guide
Bulk Loading Data
For fixed-width loads, if the input data column has fewer values than the specified column size,
COPY inserts NULL characters. The number of NULLs must match the declared column width. If
you specify a NULL string at the column level, COPY matches the string with the column width.
Note: To turn off NULLs, use the NULL AS option and specify NULL AS ''.
2. Copy the table, specifying null as 'N', and enter some data:
=> copy fw from STDIN fixedwidth colsizes(2,2) null as 'N' no commit;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> NN12
>> 23NN
>> NNNN
>> nnnn
>> \.
Administrator's Guide
Bulk Loading Data
2. Copy table fw, specifying two 2-byte column sizes, and specifying a comma (,) as the record
terminator:
=> copy fw from STDIN fixedwidth colsizes(2,2) record terminator ',';
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 1234,1444,6666
>> \.
The SELECT output indicates only two values. COPY rejected the third value (6666) because it was
not followed by a comma (,) record terminator. Fixed-width data requires a trailing record terminator
only if you explicitly specify a record terminator explicitly.
Administrator's Guide
Bulk Loading Data
Skips the total number (integer) of bytes from the input data.
SKIP records
This example uses SKIP BYTES to skip 10 bytes when loading a fixed-width table with two columns
(4 and 6 bytes):
1. Copy a table, using SKIP BYTES to skip 10 bytes of input data:
=> copy fw from stdin fixedwidth colsizes (4,6) SKIP BYTES 10;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 2222666666
>> 1111999999
>> 1632641282
>> \.
The select output indicates that COPY skipped the first 10 bytes of load data, as directed.
This example uses SKIP when loading a fixed-width (4,6) table to skip one (1) record of input data:
1. Copy a table, using SKIP to skip 1 record of input data:
=> copy fw from stdin fixedwidth colsizes (4,6) SKIP 1;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 2222666666
>> 1111999999
>> 1632641282
>> \.
Administrator's Guide
Bulk Loading Data
The SELECT output indicates that COPY skipped the first record of load data, as directed.
Administrator's Guide
Bulk Loading Data
Datatype
Padding
Integer
Bool
Float
[var]Binary
[Var]Char
DateInterval
Time
Timestamp
TimestampTZ
TimeTZ
None. All characters are significant. The COPY statement uses an internal
Date (formatted)
Use the COPY FORMAT option string to match the expected column length.
Numerics
Transform source data before loading it into the target table. Using SQL operators or functions,
you can combine two or more source columns into one column; you can also split one column
into multiple columnsfor example, split date strings into day, month, and year components and
load these into target table columns. For more information, see Transforming Data During Loads
Syntax
A COPY statement can specify multiple filler columns, where each filler column is specified by its
own FILLER parameter:
filler-id FILLER data-type
where filler-id:
Administrator's Guide
Bulk Loading Data
Note: Define the FILLER parameter data type so it is compatible with the source data. For
example, be sure to define a VARCHARso its length can contain all source data; otherwise, data
is liable to be truncated.
Usage
The FILLER parameter identifies a column of source data that the COPY command can ignore, or use
to compute new values that are loaded into the target table. For example, the following source file
contains three columns:
Sawyer|Annie|P
Finn |Harry|S
Clark|Nessa|C
Phelps|Sally|G
Phelps|Silas|J
Wilks|Peter|M
Using the FILLER parameter, you can handle this mismatch of source and target columns in two
ways:
l
Administrator's Guide
Bulk Loading Data
Administrator's Guide
Bulk Loading Data
To insert a row into the fact table of a pre-join projection, the associated values of the dimension
table's columns must be looked up. Thus, an insert into a pre-join projection shares some of the
qualities of a query. The following sections describe the behaviors associated with loading data into
pre-join projections.
Administrator's Guide
Bulk Loading Data
The following sequence of commands generate a missing foreign key error that results in a rollback
because the reference is to a non-existent dimension row.
INSERT INTO Employees (EmployeeID, First, Last, PlanID) VALUES (1004, 'Ben', 'Smith',
04);
The following sequence of commands generate a foreign key error that results in a rollback because
a duplicate row in the HealthPlans dimension table is referenced by an insert in the Employees fact
table. The error occurs when the Employees fact table references the HealthPlans dimension table.
INSERT INTO HealthPlan VALUES(02, 'MyPlan', 'PPO');
INSERT INTO Employee VALUES(1005, 'Juan', 'Hernandez', 02);
Success
Administrator's Guide
Bulk Loading Data
Session A
INSERT INTO EMPLOYEES
VALUES
(1, 'Bob');
Descriptio
n
Session B
(EMPNO, NAME)
Records
loaded by
this COPY
command
all refer to
Bob's
sales
(SELLER
= 1)
INSERT INTO EMPLOYEES (EMPNO, NAME)
VALUES
(2,'Frank');
7234 | 1
Records
loaded by
this COPY
command
all refer to
Frank's
sales
(SELLER
= 2).
COMMIT;
COMMIT;
Both
transaction
s are
successfu
l.
Failure
Session A
Session B
Description
Administrator's Guide
Bulk Loading Data
Session A
Session B
Description
2 | Terry
The transaction in
Session B fails because
the value inserted into the
dimension table in
Session A was not visible
before the COPY into the
pre-join in Session B was
initiated.
Use the nodename parameter with each file to load files from different nodes.
The single multi-node COPY parameters (nodename | ON ANY NODE) are supported only using
vsql.
Load files can be of different formats, such as BZIP, GZIP, and others. However, since file
compression is a filter, you cannot split compressed files into concurrent load streams for faster
loading.
Administrator's Guide
Bulk Loading Data
HP Vertica functions
Administrator's Guide
Bulk Loading Data
exampledb_
The LOAD_STREAMS system table includes stream names for every COPY statement that takes
more than 1-second to run. The 1-second duration includes the time to plan and execute the
statement.
HP Vertica maintains system table metrics until they reach a designated size quota (in kilobytes).
The quota is set through internal processes and cannot be set or viewed directly.
ACCEPTED_ROW_COUNT
REJECTED_ROW_COUNT
PARSE_COMPLETE_PERCENT
SORT_COMPLETE_PERCENT
Administrator's Guide
Bulk Loading Data
When a COPY statement using the DIRECT option is in progress, the ACCEPTED_ROW_COUNT value can
increase to the maximum number of rows in the input file as the rows are being parsed.
If COPY reads input data from multiple named pipes, the PARSE_COMPLETE_PERCENT value will
remain at zero (0) until all named pipes return an EOF. While COPY awaits an EOF from multiple
pipes, it can appear to be hung. However, before canceling the COPY statement, check your
system CPU and disk accesses to see if any activity is in progress.
In a typical load, the PARSE_COMPLETE_PERCENT value can either increase slowly to 100%, or jump
to 100% quickly if you are loading from named pipes or STDIN, while SORT_COMPLETE_PERCENT is
at 0. Once PARSE_COMPLETE_PERCENT reaches 100%, SORT_COMPLETE_PERCENT increases to
100%. Depending on the data sizes, a significant lag can occur between the time PARSE_
COMPLETE_PERCENT reaches 100% and the time SORT_COMPLETE_PERCENT begins to increase.
This example sets the VSQL expanded display, and then selects various columns of data from the
LOAD_STREAMS system table:
=> \pset expanded
Expanded display is on.
=> SELECT stream_name, table_name, load_start, accepted_row_count,
rejected_row_count, read_bytes, unsorted_row_count, sorted_row_count,
sort_complete_percent FROM load_streams;
-[RECORD 1 ]----------+--------------------------stream_name
| fact-13
table_name
| fact
load_start
| 2010-12-28 15:07:41.132053
accepted_row_count
| 900
rejected_row_count
| 100
read_bytes
| 11975
input_file_size_bytes | 0
parse_complete_percent | 0
unsorted_row_count
| 3600
sorted_row_count
| 3600
sort_complete_percent | 100
Incorrect data types for the table into which data is being loaded
Administrator's Guide
Bulk Loading Data
Missing delimiters
Several optional parameters let you determine how strictly the COPY statement handles rejections
when parsing data. For example, you can have COPY fail when it rejects a single row, or let
COPYallow a maximum number of parsing rejections before the load fails. This section describes
how and where rejected data us saved. It also presents theparameters to determine how
COPYhandles rejected data.
You can optionally save COPY rejections and exceptions in two other ways:
l
Use the REJECTED DATA reject_file and EXCEPTIONS except_file parameters to save one,
or both, files to a location of your choice. The reject_file includes rejected rows, while the
exceptions file contains a description of why each row was rejected.
Use the REJECTED DATA AS TABLE reject_table clause. Saving rejected rows to a reject_
table also retains the exception descriptions.
Note: You can save rejected data and exceptions to a table, or to non-default file locations, but
not to both.
The rejected table data is automatically saved in the RejectionTableData sudirectory of the data
location for your database. For example, in a VMart database installation, the rejected data records
are stored in the RejectionTableData directory as follows:
=> cd v_mart_node003_data\RejectionTableData\
=> ls
TABLE_REJECTED_RECORDS_"bg"_mytest01.verticacorp.-25441:0x6361_45035996273805099_1.1
TABLE_REJECTED_RECORDS_"bg"_mytest01.verticacorp.-25441:0x6361_45035996273805113_2.2
Administrator's Guide
Bulk Loading Data
.
.
.
TABLE_REJECTED_RECORDS_"delimr"_mytest01.verticacorp.-5958:0x3d47_45035996273815749_1.1
TABLE_REJECTED_RECORDS_"delimr"_mytest01.verticacorp.-5958:0x3d47_45035996273815749_1.2
The storage location must have been created with the USER usage type (see CREATE
LOCATION).
The user must already have been granted access to the storage location where the files exist, as
described in GRANT (Storage Location).
Administrator's Guide
Bulk Loading Data
When the number of rejected records will be greater than the REJECTMAX value (REJECTMAX+1), the
load fails. If you do not specify a value for REJECTMAX, or if the value is 0, COPY allows an unlimited
number of exceptions to occur.
Rejects the offending row and loads other rows into a table
Note: If you specify ABORT ON ERROR with the COPY statement, the load automatically rolls
back if COPY cannot parse any row.
The following table summarizes the difference between a rejected row or rollback.
Rejected Rows
following:
of these conditions:
Missing fields
Missing delimiters
constraints
l
This example illustrates what happens when HP Vertica cannot coerce a row to the requested data
type. For example, in the following COPY statement, "a::INT + b::INT" is a SQL expression in
which a and b are derived values:
Administrator's Guide
Bulk Loading Data
Vertica Analytic Database cannot parse the row to the requested data type and rejects the row:
ERROR 2827:
If a resolved to 'cat' and b to 'dog', the expression 'cat'::INT + 'dog'::INT would return an
error and the COPYstatement would fail (roll back)without loading any data.
=> SELECT 'cat'::INT + 'dog'::INT;
ERROR 3681: Invalid input syntax for integer: "cat"
The following statement would also roll back because Vertica Analytic Database cannot parse the
row to the requested data type:
=> COPY t (a FILLER VARCHAR, i AS a::INT) FROM STDIN;
In the following COPY statement, Vertica Analytic Database rejects only the offending row without
rolling back the statement. Instead of evaluating the 'cat' row as a VARCHAR type,COPY parses
'cat' directly as an INTEGER.
=> COPY t (a FILLER INT, i AS a) FROM STDIN;
In this statement, the load parser (unlike the expression evaluator) rejects the row if it contains a
field that cannot be parsed to the requested data type.
catalog_dir
Administrator's Guide
Bulk Loading Data
tablename-filename-of-source
-copy-from-rejections
The file suffix appended to the table and source file name
Once a rejections file exists, you can review its contents to resolve any load problems and reload
the data. If you save rejected data to a table, you can query the table to see both exceptions and the
rejected data.
If copying from STDIN, the filename-of-source is STDIN.
Note: You can use specific rejected data and exceptions files with one or more of the files you
are loading. Separate consecutive rejected data and exception file names with a comma (,) in
the COPY statement. Do not use the ON ANY NODE option with rejected data and exceptions
files, because ON ANY NODE is applicable only to the load file.
When you load multiple input files, you must specify a file name in the path. Keep in mind that long
input file names, combined with rejected data file names, can exceed the operating system's
maximum length (typically 255 characters). To work around file names exceeding the maximum
length, use a path for the rejected data file that differs from the default path; for example,
\tmp\<shorter-file-name>.
For all data loads (except for COPY LOCAL), COPY behaves as follows:
No rejected data file specified...
For one data file, the path is interpreted as a file, and COPY
stores all rejected data in this file. If the path is not a file,
default directory.
For multiple data files, all
default directory.
Rejected data files are not shipped to the initiator node.
Only one path per node is accepted. If more than one is
provided, COPY returns an error.
Administrator's Guide
Bulk Loading Data
catalog_dir
tablename-filename- of-source
-copy-from-exceptions
The file suffix appended to the table and source file name
Note: Using REJECTED DATA ASTABLE r_table is mutually exclusive with using the
EXCEPTIONS filename parameter. The rejected data table includes a column with the
exceptions messages. COPY does not permit both parameters. Trying to do so results in this
error:
ERROR 0: Cannot specify both an exceptions file and a rejected table in the same
statement
The optional EXCEPTIONS parameter lets you specify a file of your choice to which COPY writes load
exceptions. The EXCEPTIONS file indicates the input line number and the reason for each data
record exception in this format:
COPY: Input record number in <pathofinputfile> has been rejected (reason). Please see
pathtorejectfile, record recordnum for the rejected record.
Administrator's Guide
Bulk Loading Data
For one data file, the path is treated as a file, and COPY
default directory.
Exceptions files are not stored on the initiator node.
You can specify only one path per node. If you specify more
than one path per node, COPY returns an error.
Support SELECTstatements
To make the data in a rejected table K-safe, you can do one of the following:
l
Administrator's Guide
Bulk Loading Data
Create a table to store rejected records, and run INSERT..SELECT operations into the new table
Rejected data tables include both rejected data and the reason for the rejection (exceptions), along
with other data columns, described next. HP Vertica suggests that you periodically drop any
rejected data tables that you no longer require.
Administrator's Guide
Bulk Loading Data
>>
>>
>>
>>
2
3
a
\.
Data Type
Description
node_name
VARCHAR
file_name
VARCHAR
session_id
VARCHAR
transaction_id
INTEGER
Administrator's Guide
Bulk Loading Data
statement_id
INTEGER
batch_number
INTEGER
row_number
INTEGER
rejected_data
LONG VARCHAR
rejected_data_orig_length
INTEGER
rejected_reason
VARCHAR
2. Copy data directly into the table, using a table to store rejected data:
Administrator's Guide
Bulk Loading Data
After a file exists, you can fix load errors and use the corrected file as load input to the COPY
statement.
Administrator's Guide
Bulk Loading Data
or EXCEPTIONS parameters, rejected data and exceptions files are written to the default location, the
database catalog subdirectory, CopyErrorLogs, on each node:
\set
\set
\set
\set
\set
Note: Always use the COPY statement REJECTED DATA and EXCEPTIONS parameters to save
load rejections, or the REJECTED DATA AS TABLE clause. Using the RETURNREJECTED
parameter is supported only for internal use by the JDBC and ODBC drivers. HP Vertica's
internal-use options can change without notice.
The storage location must have been created (or altered) with the USER option (see CREATE
LOCATION and ALTER_LOCATION_USE)
The user must already have been granted READ access to the storage location where the file(s)
exist, as described in GRANT (Storage Location)
Both parameters also have an optional ON nodename clause that uses the specified path:
...[EXCEPTIONS 'path' [ON nodename ] [, ...] ]...[REJECTED DATA 'path' [ON nodename ]
[, ...] ]
While 'path' specifies the location of the rejected data and exceptions files (with their corresponding
parameters), the optional ON nodename clause moves any existing rejected data amd exception
files on the node to the specified path on the same node.
Administrator's Guide
Bulk Loading Data
This example uses variables to specify exception and rejected date files (except_s2 and reject_
s2) on a remote node. The COPY statement consists of a single input file on a remote node
(site02):
\set remote_dir /vertica/test_dev/tmp_ms/\set except_s2 '''':remote_dir'exceptions'''
\set reject_s2 '''':remote_dir'rejections'''
COPY large_tbl FROM :file1 ON site02DELIMITER '|'
REJECTED DATA :reject_s2 ON site02
EXCEPTIONS :except_s2 ON site02;
This example uses variables to specify that the exception and rejected data files are on a remote
node (indicated by :remote_dir). The inputs to the COPY statement consist of multiple data files
Administrator's Guide
Bulk Loading Data
on two nodes (site01 and site02). The exceptions and rejected data options use ON
nodename clause with the vsql variables to indicate where the files reside (site01 and site02):
\set
\set
\set
\set
\set
Administrator's Guide
Bulk Loading Data
2. Specify two different file names for the two options. You cannot use one file for both the
REJECTED DATA and the EXCEPTIONS.
3. When you invoke COPY LOCAL or COPY LOCAL FROM STDIN, the files you specify need
not pre-exist. If they do, COPY LOCAL must be able to overwrite them.
You can specify the path and file names with vsql variables:
\set rejected ../except_reject/copyLocal.rejected
\set exceptions ../except_reject/copyLocal.exceptions
Note: Using COPYLOCALdoes not support storing rejected data in a table, as you can when
using the COPY statement.
When you use the COPY LOCAL or COPY LOCAL FROM STDIN statement, specify the variable
names for the files with their corresponding parameters:
=> COPY large_tbl FROM LOCAL rejected data :rejected exceptions :exceptions;
=> COPY large_tbl FROM LOCAL STDIN rejected data :rejected exceptions :exceptions;
See Also
l
COPY
ANALYZE_CONSTRAINTS
Administrator's Guide
Trickle Loading Data
Allows users to see the most recently committed data without holding any locks.
DELETE does not actually delete data from disk storage; it marks rows as deleted so that they
exist for historical queries.
UPDATE writes two rows: one with new data and one marked for deletion.
Like COPY, by default, INSERT, UPDATE and DELETE statements write the data to the WOS.
On overflow, the statements write data to the ROS. For large INSERTS or UPDATES, use the
DIRECT keyword to force HP Vertica to write rows directly to ROS. Loading large number of rows
as single row inserts is not recommended for performance reasons. Use the COPY statement,
instead.
Administrator's Guide
Trickle Loading Data
Administrator's Guide
Copying and Exporting Data
EXPORT TO VERTICA
To execute either of these statements requires first creating a connection to the other HP Vertica
database.
EXPORT_CATALOG
EXPORT_OBJECTS
EXPORT_TABLES
While copying and exporting data is similar to Backing Up and Restoring the Database, you should
use them for different purposes, outlined below:
Task
Backup and
Restore
YES
NO
YES
YES
YES
NO
changes
Manage database objects (a single table or selected
table rows)
Use external locations to back up and restore your
database
Administrator's Guide
Copying and Exporting Data
Task
Backup and
Restore
NO
YES
YES
NO
NO
YES
database
Use SQL commands to incorporate copy and export
tasks into DB operations
The following sections explain how you import and export data between HP Vertica databases.
When importing from or exporting to an HP Vertica database, you can connect only to a database
that uses trusted- (username-only) or password-based authentication, as described in
Implementing Security. Neither LDAP nor SSL authentication is supported.
Exporting Data
You can export data from an earlier HP Vertica release, as long as the earlier release is a version of
the last major release. For instance, for Version 6.x, you can export data from any version of 5.x,
but not from 4.x.
You can export a table, specific columns in a table, or the results of a SELECT statement to
another HP Vertica database. The table in the target database receiving the exported data must
already exist, and have columns that match (or can be coerced into) the data types of the columns
you are exporting.
Exported data is always written in AUTO mode.
When you load data using AUTO mode, HP Vertica inserts the data first into the WOS. If the WOS
is full, then HP Vertica inserts the data directly into ROS. See the COPY statement for more
details.
Exporting data fails if either side of the connection is a single-node cluster installed to localhost or
you do not specify a host name or IP address.
Exporting is a three-step process:
1. Use the CONNECT SQL statement to connect to the target database that will receive your
exported data.
Administrator's Guide
Copying and Exporting Data
2. Use the EXPORT SQL statement to export the data. If you want to export multiple tables or the
results of multiple SELECT statements, you need to use multiple EXPORT statements. All
statements will use the same connection to the target database.
3. When you are finished exporting data, use the DISCONNECT SQL statement to disconnect
from the target database.
See the entries for CONNECT, EXPORT, and DISCONNECT statements in the SQL Reference
Manual for syntax details.
If source and destination tables have an Identity column, you do not need to list them.
If source has an Identity column, but not the destination, specify both the source and destination
columns.
Note: In earlier releases, Identity columns were ignored. Now, failure to list which Identity
columns to export can cause an error, because the Identity column is not ignored and will be
interpreted as missing in the destination table.
The default behavior for EXPORT TO VERTICA is to let you export Identity columns by specifying
them directly in the source table. To disable this behavior globally, set the
CopyFromVerticaWithIdentity configuration parameter, described in Configuration Parameters.
Administrator's Guide
Copying and Exporting Data
--------------23416
(1 row)
The following statement demonstrates exporting a portion of a table using a simple SELECT
statement.
=> EXPORT TO VERTICA testdb.ma_customers AS SELECT customer_key, customer_name, annual_
income
-> FROM customer_dimension WHERE customer_state = 'MA';
Rows Exported
--------------3429
(1 row)
This statement exports several columns from one table to several different columns in the target
database table using column lists. Remember that when supplying both a source and destination
column list, the number of columns must match.
=> EXPORT TO VERTICA testdb.people (name, gender, age) FROM customer_dimension
-> (customer_name, customer_gender, customer_age);
Rows Exported
--------------23416
(1 row)
You can export tables (or columns) containing Identity and Auto-increment values, but the
sequence values are not incremented automatically at their destination.
You can also use the EXPORT TO VERTICA statement with a SELECT AT EPOCH LATEST
expression to include data from the latest committed DML transaction.
Disconnect from the database when the export is complete:
=> DISCONNECT testdb;
DISCONNECT
Note: Closing your session also closes the database connection. However, it is a good
practice to explicitly close the connection to the other database, both to free up resources and
to prevent issues with other SQL scripts you may run in your session. Always closing the
connection prevents potential errors if you run a script in the same session that attempts to
open a connection to the same database, since each session can only have one connection to
a particular database at a time.
Administrator's Guide
Copying and Exporting Data
Copying Data
You can import a table or specific columns in a table from another HP Vertica database. The table
receiving the copied data must already exist, and have columns that match (or can be coerced into)
the data types of the columns you are copying from the other database. You can import data from
an earlier HP Vertica release, as long as the earlier release is a version of the last major release. For
instance, for Version 6.x, you can import data from any version of 5.x, but not from 4.x.
Importing and exporting data fails if either side of the connection is a single-node cluster installed to
localhost, or you do not specify a host name or IP address.
Importing is a three-step process:
1. Use the CONNECT SQL statement to connect to the source database containing the data you
want to import.
2. Use the COPY FROM VERTICA SQL statement to import the data. If you want to import
multiple tables, you need to use multiple COPY FROM VERTICA statements. They all use the
same connection to the source database.
3. When you are finished importing data, use the DISCONNECT SQL statement to disconnect
from the source database.
See the entries for CONNECT, COPY FROM VERTICA, and DISCONNECT statements in the
SQL Reference Manual for syntax details.
If source and destination tables have an Identity column, you do not need to list them.
If source has an Identity column, but not the destination, specify both the source and destination
columns.
Note: In earlier releases, Identity columns were ignored. Now, failure to list which Identity
columns to export can cause an error, because the Identity column is not ignored and will be
interpreted as missing in the destination table.
After importing the columns, the Identity column values do not increment automatically. Use
ALTER SEQUENCE to make updates.
Administrator's Guide
Copying and Exporting Data
The default behavior for this statement is to import Identity (and Auto-increment) columns by
specifying them directly in the source table. To disable this behavior globally, set the
CopyFromVerticaWithIdentity configuration parameter, described in Configuration Parameters.
Examples
This example demonstrates connecting to another database, copying the contents of an entire table
from the source database to an identically-defined table in the current database directly into ROS,
and then closing the connection.
=> CONNECT TO VERTICA vmart USER dbadmin PASSWORD '' ON 'VertTest01',5433;
CONNECT
=> COPY customer_dimension FROM VERTICA vmart.customer_dimension DIRECT;
Rows Loaded
------------500000
(1 row)
=> DISCONNECT vmart;
DISCONNECT
This example demonstrates copying several columns from a table in the source database into a
table in the local database.
=> CONNECT TO VERTICA vmart USER dbadmin PASSWORD '' ON 'VertTest01',5433;
CONNECT
=> COPY people (name, gender, age) FROM VERTICA
-> vmart.customer_dimension (customer_name, customer_gender,
-> customer_age);
Rows Loaded
------------500000
(1 row)
=> DISCONNECT vmart;
DISCONNECT
You can copy tables (or columns) containing Identity and Auto-increment values, but the sequence
values are not incremented automatically at their destination.
Administrator's Guide
Copying and Exporting Data
By default, importing from and exporting to another HP Vertica database uses the private network.
Note: Ensure port 5433 or the port the HP Vertica database is using is not blocked.
To use the public network address for copy and export activities, configure the system to use the
public network to support exporting to or importing from another HP Vertica cluster:
l
Use the CREATE SUBNET statement provide your subnet with a name and to identify the
subnet routing prefix.
After you've identified the subnet or network interface to be used for import/export, you must
Identify the Database Or Nodes Used For Import/Export.
Administrator's Guide
Copying and Exporting Data
See Also
l
CREATE SUBNET
ALTER SUBNET
DROP SUBNET
Use the ALTER DATABASE statement to specify the subnet name of the public network. When
you do so, all nodes in the database will automatically use the network interface on the subnet
for import/export operations.
Use the ALTER NODE statement to specify the network interface of the public network on each
individual node.
See Also
l
ALTERDATABASE
CREATESUBNET
V_MONITOR_NETWORK INTERFACES
Administrator's Guide
Copying and Exporting Data
To recreate...
EXPORT_
CATALOG
l
EXPORT_
Non-virtual objects up to, and including, the schema of one or more tables.
TABLES
EXPORT_
OBJECTS
The designs and object definitions that the script creates depend on the EXPORT_ function scope
you specify. The following sections give examples of the commands and output for each function
and the scopes it supports.
Administrator's Guide
Copying and Exporting Data
If you specify the destination argument as an empty string (''), the function writes the export
results to STDOUT.
Note: A superuser can export all available database output to a file with the EXPORT_
functions. For a non-superuser, the EXPORT_ functions generate a script containing only the
objects to which the user has access.
DESIGN
(This is the default scope.)
DESIGN ALL
TABLES
Function Summary
Here is the function syntax, described in EXPORT_CATALOG in the SQL Reference Manual:
EXPORT_CATALOG (['destination'] , ['scope'] )
Administrator's Guide
Copying and Exporting Data
=> SELECTEXPORT_CATALOG('/home/dbadmin/xtest/sql_cat_design.sql','DESIGN');
EXPORT_CATALOG
------------------------------------Catalog data exported successfully
(1 row)
The SQL script includes the following types of statements, each needed to recreate a new
database:
l
CREATE SCHEMA
CREATE TABLE
CREATE VIEW
CREATE SEQUENCE
PARTITION BY
Projection Considerations
If a projection to export was created with no ORDER BY clause, the SQL script reflects the default
behavior for projections. HP Vertica implicitly creates projections using a sort order based on the
SELECT columns in the projection definition. The EXPORT_CATALOG script reflects this
behavior.
The EXPORT_CATALOG script is portable as long as all projections were generated using
UNSEGMENTED ALL NODES or SEGMENTED ALL NODES.
Administrator's Guide
Copying and Exporting Data
CREATE SCHEMA
CREATE TABLE
CREATE SEQUENCE
See Also
l
EXPORT_CATALOG
EXPORT_OBJECTS
EXPORT_TABLES
Exporting Tables
Exporting Objects
Administrator's Guide
Copying and Exporting Data
Exporting Tables
Use the EXPORT_TABLES function to recreate one or more tables, and related objects, on a
different cluster. Specify one of the following options to determine the scope:
To export...
constraints.
One or more named objects, such as tables or sequences in one or
A comma-delimited list of
items:
'myschema.newtable,
yourschema.oldtable'
Function Syntax
EXPORT_TABLES (['destination' ] , ['scope' ] )
Administrator's Guide
Copying and Exporting Data
The SQL script includes the following types of statements, depending on what is required to
recreate the tables and any related objects (such as sequences):
l
CREATE SCHEMA
CREATE TABLE
CREATE SEQUENCE
PARTITION BY
The SQL script includes the following types of statements, depending on what is required to create
the list of objects:
l
CREATE SCHEMA
CREATE TABLE
CREATE SEQUENCE
Administrator's Guide
Copying and Exporting Data
Following are the contents of the export_one_sequence.sql output file using a more command:
[dbadmin@node01 xtest]$more export_one_sequence.sql
CREATE SEQUENCE public.my_seq ;
Exporting Objects
Use EXPORT_OBJECTS function to recreate the exported objects. Specify one of the following
options to determine the scope:
To export...
constraints.
One or more named objects, such as tables or views in one or
A comma-delimited list of
items:
'myschema.newtable,
yourschema.oldtable'
Function Syntax
EXPORT_OBJECTS(['destination'
] , ['scope'
] , [
'ksafe' ] )
Administrator's Guide
Copying and Exporting Data
CREATE SCHEMA
CREATE TABLE
CREATE VIEW
CREATE SEQUENCE
PARTITION BY
Here is a snippet from the start of the output SQL file, and the end, showing the KSAFE statement:
CREATE SCHEMA store;CREATE SCHEMA online_sales;
CREATE SEQUENCE public.my_seq ;
CREATE TABLE public.customer_dimension
(
customer_key int NOT NULL,
customer_type varchar(16),
customer_name varchar(256),
customer_gender varchar(8),
title varchar(8),
household_id int,
.
.
.
);
.
Administrator's Guide
Copying and Exporting Data
.
.
SELECT MARK_DESIGN_KSAFE(0);
CREATE SCHEMA
CREATE TABLE
CREATE VIEW
CREATE SEQUENCE
If you specify a view without its dependencies, the function displays a WARNING. The SQL script
includes a CREATE statement for the dependent object, but will be unable to create it without the
necessary relations:
VMart=> select export_objects('nameObjectsList', 'test2, tt, my_seq, v2' );
WARNING 0: View public.v2 depends on other relations
export_objects
------------------------------------Catalog data exported successfully
(1 row)
Here are the contents of the output file of the example, showing the sample table test7 and the v1
view:
Administrator's Guide
Copying and Exporting Data
Administrator's Guide
Bulk Deleting and Purging Data
Description
DROP
TABLE
DELETE
Marks rows with delete vectors and stores them so data can be rolled back to a
FROM
previous epoch. The data must eventually be purged before the database can
TABLE
TRUNCATE Removes all storage and history associated with a table. The table structure is
TABLE
preserved for future use. The results of this command cannot be rolled back.
DROP_
Removes one partition from a partitioned table. Each partition contains a related
PARTITION
subset of data in the table. Partitioned data can be dropped efficiently, and
provides query performance benefits. See Using Table Partitions.
The following table provides a quick reference for the different delete operations you can use. The
"Saves History" column indicates whether data can be rolled back to an earlier epoch and queried at
a later time.
Syntax
Performanc
e
Commit
s Tx
Saves
Histor
y
Normal
No
Yes
High
No
No
Normal
No
Yes
Normal
No
Yes
Normal
No
Yes
High
Yes
No
DROP base_table
High
Yes
No
TRUNCATE base_table
High
Yes
No
Administrator's Guide
Bulk Deleting and Purging Data
Syntax
Performanc
e
Commit
s Tx
Saves
Histor
y
TRUNCATE temp_table
High
Yes
No
DROP PARTITION
High
Yes
No
To delete both table data and definitions and start from scratch, use the DROP TABLE
[CASCADE] command.
To drop data, while preserving table definitions so that you can quickly and easily reload data,
use TRUNCATE TABLE. Note that unlike DELETE, TRUNCATE does not have to mark each
row with delete vectors, so it runs much more quickly.
To perform bulk delete operations on a regular basis, HP Vertica recommends using Partitioning.
To perform occasional small deletes or updates with the option to roll back or review history, use
DELETE FROM TABLE. See Best Practices for DELETE and UPDATE.
For details on syntax and usage, see DELETE, DROP TABLE, TRUNCATE TABLE, CREATE
TABLE and DROP_PARTITION in the SQL Reference Manual.
Administrator's Guide
Bulk Deleting and Purging Data
Query performance after large deletesA large number of (unpurged) deleted rows can
negatively affect query performance.
To eliminate rows that have been deleted from the result, a query must do extra processing. If
10% or more of the total rows in a table have been deleted, the performance of a query on the
table degrades. However, your experience may vary depending on the size of the table, the table
definition, and the query. If a table has a large number of deleted rows, consider purging those
rows to improve performance. For more information on purging, see Purging Deleted Data.
Recovery performanceRecovery is the action required for a cluster to restore K-safety after
a crash. Large numbers of deleted records can degrade the performance of a recovery. To
improve recovery performance, purge the deleted rows. For more information on purging, see
Purging Deleted Data.
ConcurrencyDELETE and UPDATE take exclusive locks on the table. Only one DELETE or
UPDATE transaction on a table can be in progress at a time and only when no loads (or INSERTs)
are in progress. DELETEs and UPDATEs on different tables can be run concurrently.
Pre-join projectionsAvoid pre-joining dimension tables that are frequently updated. DELETE
and UPDATE operations on pre-join projections cascade to the fact table, causing large DELETE or
UPDATE operations.
For detailed tips about improving DELETE and UPDATE performance, see Optimizing DELETEs and
UPDATEs for Performance.
Administrator's Guide
Bulk Deleting and Purging Data
Caution: HP Vertica does not remove deleted data immediately but keeps it as history for the
purposes of historical query. A large amount of history can result in slower query
performance. For information about how to configure the appropriate amount of history to
retain, see Purging Deleted Data.
In the following query, both p1 and p2 are eligible for DELETE and UPDATE optimization because
column a is available:
DELETE from t WHERE a = 1;
In the following example, only projection p1 is eligible for DELETE and UPDATE optimization because
the b column is not available in p2:
DELETE from t WHERE b = 1;
Administrator's Guide
Bulk Deleting and Purging Data
The first projection references all columns in tb1 and sorts on column a:
CREATE PROJECTION tb1_p AS SELECT a, b, c, d FROM tb1 ORDER BY a;
Consider the following DML statement, which references tb1.a in its WHERE clause. Since both
projections on tb1 contain column a, both are eligible for the optimized DELETE:
DELETE FROM tb1 WHERE tb1.a IN (SELECT tb2.i FROM tb2);
Restrictions
Optimized DELETEs are not supported under the following conditions:
l
With replicated and pre-join projections if subqueries reference the target table. For example, the
following syntax is not supported:
DELETE FROM tb1 WHERE tb1.a IN (SELECT e FROM tb2, tb2 WHERE tb2.e = tb1.e);
With subqueries that do not return multiple rows. For example, the following syntax is not
Administrator's Guide
Bulk Deleting and Purging Data
supported:
DELETE FROM tb1 WHERE tb1.a = (SELECT k from tb2);
To optimize the DELETEs, make time_key appear in the ORDER BY clause of all your projections.
This schema design results in better performance of the DELETE operation.
In addition, add additional sort columns to the sort order such that each combination of the sort key
values uniquely identifies a row or a small set of rows. For more information, see Choosing Sort
Order: Best Practices. To analyze projections for sort order issues, use the EVALUATE_DELETE_
PERFORMANCE function.
Administrator's Guide
Bulk Deleting and Purging Data
A purge operation permanently removes deleted data from physical storage so that the disk space
can be reused. HP Vertica gives you the ability to control how much deleted data is retained in the
physical storage used by your database by performing a purge operation using one of the following
techniques:
l
Both methods set the Ancient History Mark (AHM), which is an epoch that represents the time
until which history is retained. History older than the AHM are eligible for purge.
Note: Large delete and purge operations in HP Vertica could take a long time to complete, so
use them sparingly. If your application requires deleting data on a regular basis, such as by
month or year, HP recommends that you design tables that take advantage of table
partitioning. If partitioning tables is not suitable, consider the procedure described in Rebuilding
a Table. The ALTER TABLE..RENAME command lets you build a new table from the old table,
drop the old table, and rename the new table in its place.
Administrator's Guide
Bulk Deleting and Purging Data
seconds is the amount of time (in seconds) for which to save deleted data.
-1 indicates that you do not want to use the HistoryRetentionTime configuration parameter to
determine which deleted data is eligible to be purged. Use this setting if you prefer to use the
other method (HistoryRetentionEpochs) for determining which deleted data can be purged.
The following example sets the history epoch retention level to 240 seconds:
=> ALTERDATABASEmydb SET HistoryRetentionTime = 240;
2. Set the history epoch retention level through the HistoryRetentionEpochs configuration
parameter:
Administrator's Guide
Bulk Deleting and Purging Data
The following example sets the number of historical epochs to save to 40:
=> ALTERDATABASE mydb SET HistoryRetentionEpochs = 40;
Modifications are immediately implemented across all nodes within the database cluster. You do
not need to restart the database.
Note: If both HistoryRetentionTime and HistoryRetentionEpochs are specified,
HistoryRetentionTime takes precedence.
See Epoch Management Parameters for additional details.
Disabling Purge
If you want to preserve all historical data, set the value of both historical epoch retention parameters
to -1, as follows:
=> ALTER DABABASE mydb SET HistoryRetentionTime = -1;
=> ALTERDATABASE mydb SET HistoryRetentionEpochs = -1;
SET_AHM_TIME() sets the AHM to the epoch that includes the specified TIMESTAMP
value on the initiator node.
Administrator's Guide
Bulk Deleting and Purging Data
GET_AHM_EPOCH() returns the number of the epoch in which the AHM is located.
MAKE_AHM_NOW() sets the AHM to the greatest allowable value (now), and lets you drop
pre-existing projections. This purges all deleted data.
When you use SET_AHM_TIME or GET_AHM_TIME, keep in mind that the timestamp you
specify is mapped to an epoch, which has (by default) a three-minute granularity. Thus, if you
specify an AHM time of '2008-01-01 00:00:00.00' the resulting purge could permanently
remove as much as the first three minutes of 2008, or could fail to remove the last three
minutes of 2007.
Note: The system prevents you from setting the AHM beyond the point at which it would
prevent recovery in the event of a node failure.
3. Manually initiate a purge using one of the following SQL functions (described in the SQL
Reference Manual):
n
Administrator's Guide
Managing the Database
Managing Nodes
Managing Workloads
Native connection load balancing is a feature built into the HP Vertica server and client libraries
that redirect client connections at the application level.
Internet Protocol Virtual Server (IPVS) is software that can be installed on several hosts in the
HP Vertica cluster that provides connection load balancing at the network level.
In most situations, you should use native connection load balancing instead of IPVS. The following
sections explain each option in detail and explain their benefits.
Administrator's Guide
Managing the Database
IPVS Overview
IPVS is a feature of the Linux kernel which lets a single host act as a gateway to an entire cluster of
hosts. The load balancing host creates a virtual IPaddress on the network. When a client connects
to the virtual IPaddress, the IPVS load balancer transparently redirects the connection to one of the
hosts in the cluster. For more on IPVS, see Connection Load Balancing Using IPVS.
Easy to set up. All that is required is that the client and server enable connection load balancing.
IPVS requires the configuration of several additional software packages and the network.
Easily overridden in cases where you need to connect to a specific host (for example, when
using the COPYstatement to load a file a specific host's filesystem).
Less at risk of host failures affecting connection load balancing. IPVS is limited to a master and
a backup server. All hosts in HP Vertica are capable of redirecting connections for connection
load balancing. As long as HP Vertica is running, it can perform connection load balancing.
Less memory and CPUintensive than running a separate IPVSprocess on several database
hosts.
Administrator's Guide
Managing the Database
IPVShurts load performance, since there is an additional network hop from the IPVS server to a
host in the database. Using native connection load balancing, clients connect directly to hosts in
the Vertica Analytic Database cluster.
Restrictive firewalls between the clients and the hosts in the database cluster can interfere with
native connection load balancing. In order for native connection load balancing to work, clients
must be able to access every host in the HP Vertica cluster. Any firewall between the hosts and
the clients must be configured to allow connections to the HP Vertica database. Otherwise, a
client could be redirected to a host that it cannot access. IPVS can be configured so that the
virtual IPaddress clients use to connect is outside of the firewall while the hosts can be behind
the firewall.
Since native connection load balancing works at the application level, clients must opt-in to have
their connections load balanced. Because it works at the network protocol level, IPVScan force
connections to be balanced.
How you choose to implement connection load balancing depends on your network environment.
Since native load connection balancing is easier to implement, you should use it unless your
network configuration requires that clients be separated from the hosts in the HP Vertica database
by a firewall.
Administrator's Guide
Managing the Database
1. The client connects to a host in the database cluster, with a connection parameter indicating
that it is requesting a load-balanced connection.
2. The host chooses a host from the list of currently up hosts in the database based on the current
load balancing scheme (see below).
3. The host tells the client which host it has selected to handle the client's connection.
4. If the host chose another host in the database to handle the client connection, the client
disconnects from the initial host. Otherwise, the client jumps to step 6.
5. The client establishes a connection to the host that will handle its connection. The client sets
this second connection request so that the second host does not interpret the connection as a
request for load balancing.
6. The client connection proceeds as usual, (negotiating encryption if the connection has
SSLenabled, and proceeding to authenticating the user ).
This entire process is transparent to the client application. The client driver automatically
disconnects from the initial host and reconnects to the host selected for load balancing.
Notes
l
In mixed IPv4 and IPv6 environments, balancing only works for the address family for which you
have configured native load balancing. For example, if you have configured load balancing using
an IPv4 address, then IPv6 clients cannot use load balancing, however the IPv6 clients can still
connect, but load balancing does not occur.
Native connection load balancing works with the ADO.NET driver's connection pooling. The
connection the client makes to the initial host and the final connection to the load balanced host
used pooled connections if they are available.
The native load-balancer returns an IPaddress for the client to use. This address must therefore
be one that the client can reach. If your nodes are on a private network, be sure to publish a
public address if you are using native load-balancing. See export_address in the NODES table.
For client applications using the JDBC and ODBC driver in conjunction with third-party
connection pooling solutions, the initial connection is not pooled since it is not a full client
connection. The final connection is pooled, since it is a standard client connection.
The client libraries include a failover feature that allow them to connect to backup hosts if the
host specified in the connection properties is unreachable. When using native connection load
Administrator's Guide
Managing the Database
balancing, this failover feature is only used for the initial connection to the database. If the host
to which the client was redirected does not respond to the client's connection request, the client
does not attempt to connect to a backup host and instead returns a connection error to the user.
Since clients are only redirected to hosts that are known to be up, this sort of connection failure
should only occur if the targeted host happened to go down at the same moment the client is
redirected to it. See ADO.NET Connection Failover, JDBC Connection Failover, and ODBC
Connection Failover in the Connecting to HP Vertica Guide for more information.
NONE: Disables native connection load balancing. This is the default setting.
ROUNDROBIN:Chooses the next host from a circular list of currently up hosts in the database
(i.e. node #1, node #2, node #3, etc. until it wraps back to node #1 again). Each host in the
cluster maintains its own pointer to the next host in the circular list, rather than there being a
single cluster-wide state.
RANDOM:Chooses a host at random from the list of currently up hosts in the cluster.
You set the native connection load balancing scheme using the SET_LOAD_BALANCE_POLICY
function. See Enabling and Disabling Native Connection Load Balancing for instructions.
Related Tasks
Enabling and Disabling Native Connection Load Balancing
729
731
Administrator's Guide
Managing the Database
-------------------------------------------------------------------------------Successfully changed the client initiator load balancing policy to: roundrobin
(1 row)
Note: When a client makes a connection, the native load-balancer chooses a node and returns
the value from the export_address column in the NODES table. The client then uses the
export_address to connect. The node_address specifies the address to use for inter-node
and spread communications. When a database is installed, the export_address and node_
address are set to the same value. If you installed HP Vertica on a private address, then you
must set the export_address to a public address for each node.
By default, client connections are not load balanced, even when connection load balancing is
enabled on the server. Clients must set a connection parameter to indicates they are willing to have
their connection request load balanced. See Enabling Native Connection Load Balancing in
ADO.NET, Enabling Native Connection Load Balancing in JDBC and Enabling Native Connection
Load Balancing in ODBC in the Connecting to HP Vertica Guide for more information.
Important: In mixed IPv4 and IPv6 environments, balancing only works for the address family
for which you have configured load balancing. For example, if you have configured load
balancing using an IPv4 address, then IPv6 clients cannot use load balancing, however the
IPv6 clients can still connect, but load balancing does not occur.
Administrator's Guide
Managing the Database
(1 row)
Related Information
About Native Connection Load Balancing
727
Related Tasks
Monitoring Native Connection Load Balancing
731
Related Information
About Native Connection Load Balancing
727
Related Tasks
Enabling and Disabling Native Connection Load Balancing
729
Administrator's Guide
Managing the Database
The Virtual IP (VIP): The IP address that is accessed by all client connections.
Real server IPs (RIP): The IP addresses of client network interfaces used for connecting
database clients to the database engine.
Virtual server: The single point of entry that provides access to a cluster, based on dynamic
node selection.
The IPVS load balancer is two node stand-by redundancy only. The IPVS redundancy model is
different than that of the HP Vertica Analytic Database. See Failure Recovery for details on HP
Vertica redundancy.
Client connections made through the Virtual IP (VIP) are managed by a primary (master) director
node, which is one of the real server nodes (RIP). The master director handles the routing of
requests by determining which node has the fewest connections and sending connections to that
node. If the director node fails for any reason, a failover (slave) director takes over request routing
until the primary (master) director comes back online.
For example, if a user connects to node03 in a three-node cluster and node03 fails, the current
transaction rolls back, the client connection fails, and a connection must be reestablished on
another node.
The following graphic illustrates a three-node database cluster where all nodes share a single VIP.
The cluster contains a master director (node01), a slave director (node02), and an additional host
(node03) that together provide the minimum configuration for high availability (K-safety). In this
setup (and in the configuration and examples that follow), node01 and node02 play dual roles as
IPVS directors and HP Vertica nodes.
Administrator's Guide
Managing the Database
Notes
l
Load balancing on a VIP is supported for Linux Red Hat Enterprise Linux 5 and 6, 64-bit.
HP Vertica must be installed on each node in the cluster; the database can be installed on any
node, but only one database can be running on an HP Vertica cluster at a time.
Although a 0 K-safety (two-node) design is supported, HP strongly recommends that you create
the load-balancing network using a minimum three-node cluster with K-safety set to 1. This way
if one node fails, the database stays up. See Designing for K-Safety for details.
When K-safety is set to 1, locate the IPVS master and slave on HP Vertica database nodes that
comprise a buddy projections pair. This is the best way to ensure high-availability loadbalancing. See High Availability Through Projections for details on buddy projections.
If the node that is the IPVS master fails completely, the slave IPVS takes over load balancing.
However, if the master only partially fails (i.e., it loses some of its processes but the node is still
up), you may have to modify IP addresses to direct network traffic to the slave node.
Alternatively, you can try to restart the processes on the master.
Subsequent topics in this section describe how to set up two directors (master and slave), but
you can set up more than two directors. See the Keepalived User Guide for details. See also the
Linux Virtual Server Web site.
Administrator's Guide
Managing the Database
requests in the event of a master node failure. The slave is collocated with another database
cluster node
n nodes database cluster, such as at least one failover node to provide the minimum configuration
for high availability. (K-safety).
l
Virtual IP (VIP) address (generally assigned to eth0 in Linux) is the public network interface
over which database clients connect.
Note: The VIP must be public so that clients outside the cluster can contact it.
Once you have set up an HP Vertica cluster and created a database, you can choose the nodes
that will be directors. To achieve the best high-availability load balancing result when K-safety is set
to 1, ensure that the IPVS master node and the slave node are located on HP Vertica database
nodes with a buddy projections pair. (See High Availability Through Projections for information on
buddy projections.)
The instructions in this section use the following node configuration:
Pre-configured IP
Node assignment
Public IPs
VIP
10.10.51.180
Private IPs
10.10.51.55
192.168.51.1
node02
10.10.51.56
192.168.51.2
node03
10.10.51.57
192.168.51.3
Administrator's Guide
Managing the Database
Notes
l
In the above table, the private IPs determine which node to send a request to. They are not the
same as the RIPs.
The VIP must be on the same subnet as the nodes in the HP Vertica cluster.
Both the master and slave nodes (node01 and node02 in this section) require additional
installation and configuration, as described in Configuring the Directors.
Use the command $cat /etc/hosts to display a list of all hosts in your cluster
The following external web sites might be useful. The links worked at the last date of publication,
but HP Vertica does not manage this content.
See Also
l
LVS-HOWTO Page
3. Set up the loopback adapter with an alias for the VIP by adding the following block to the end of
the file:
Administrator's Guide
Managing the Database
## vip deviceDEVICE=lo:0
IPADDR=10.10.51.180
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback
Note: When you add the above block to your file, be careful not to overwrite the
127.0.0.1 parameter, which is required for proper system operations.
Note: For additional details, refer to the LVS-HOWTO Page. You might also refer to the
Linux Virtual Server Wiki page for information on using arp_announce/arp_ignore to
disable the Address Resolution Protocol.
4. Use ifconfig to verify that the interface is on the same subnet as the VIP:
Administrator's Guide
Managing the Database
[root@node01]# /sbin/ifconfig
In the following output, the eth0 (10.10.51.55) is in the same network as the VIP
(10.10.51.180):
eth0
eth1
lo
lo:0
Administrator's Guide
Managing the Database
[root@node01]# /sbin/ifup lo
Administrator's Guide
Managing the Database
Install HP Vertica
2. Download the IPVS Load Balancer package for Red Hat Enterprise Linux 5 from the
my.vertica.com website to a location on the master server, such as to /tmp.
3. Change directory to the location of the downloaded file:
# cd /tmp
4. Install (or upgrade) the Load Balancer package using the rpm -Uvh command.
Administrator's Guide
Managing the Database
The following is an example for the Red Hat Linux package only; package names name could
change between releases:
# rpm -Uvh vertica-ipvs-load-balancer-<current-version>.x86_64.RHEL5.rpm
2. Download the IPVS Load Balancer package for Red Hat Enterprise Linux 6 from the
my.vertica.com website to a location on the master server, such as to /tmp.
3. Change directory to the location of the downloaded file:
# cd /tmp
6. Install (or upgrade) the Load Balancer package using the rpm -Uvh command.
The following is an example for the Red Hat Linux package only; package names name could
change between releases:
# rpm -Uvh vertica-ipvs-load-balancer-<current-version>.x86_64.RHEL6.rpm
Administrator's Guide
Managing the Database
For details about each of these parameters, refer to the ipvsadm(8) - Linux man page.
You must use IP addresses, not node names, or the spread.pl script could fail.
Administrator's Guide
Managing the Database
If you do not specify private interfaces, HP Vertica uses the public RIPs for the MISC check, as
shown in step 3 below.
2. Run the HP-supplied configuration script with the appropriate switches; for example:
# /sbin/configure-keepalived.pl --ripips 10.10.50.116,10.10.50.117,10.10.50.118
--priv_ips 192.168.51.116,192.168.51.117,192.168.51.118
--ripport 5433
--iface eth0
--emailto [email protected]
--emailfrom [email protected]
--mailserver mail.server.com
--master
--authpass password
--vip 10.10.51.180
--delayloop 2
--algo lc
--kind DR
--priority 100
Caution: The --authpass (password) switch must be the same on both the master and
slave directors.
3. Check the keepalived.conf file to verify private and public IP settings for the --ripips and -priv_ips switches and make sure the real_server IP address is public.
#
cat /etc/keepalived/keepalived.conf
4. Start spread:
Administrator's Guide
Managing the Database
# /etc/init.d/spread.pl start
The spread.pl script writes to the check.txt file, which is is rewritten to include only the
remaining nodes in the event of a node failure. Thus, the virtual server knows to stop sending
vsql requests to the failed node.
5. Start keepalived on node01:
# /etc/init.d/keepalived start
6. If not already started, start sendmail to allow mail messages to be sent by the directors:
# /etc/init.d/sendmail start
7. Repeat steps 1-5 on the slave director (node02), using the same switches, except
(IMPORTANT) replace the --master switch with the --slave switch.
Note: Tip: Use a lower priority for the slave --priority switch. HP currently suggests 50.
# /sbin/configure-keepalived.pl
--ripips 10.10.50.116,10.10.50.117,10.10.50.118
--priv_ips 192.168.51.116,192.168.51.117,192.168.51.118
--ripport 5433
--iface eth0
--emailto [email protected]
--emailfrom [email protected]
--mailserver mail.server.com
--slave
--authpass password
--vip 10.10.51.180
--delayloop 2
--algo lc
--kind DR
--priority 100
See Also
l
Administrator's Guide
Managing the Database
To verify connection distribution over multiple nodes, repeat the following statement multiple times
and observe connection distribution in an lc (least amount of connections) fashion.
$
Replace <VIP> in the above script with the IP address of your virtual server; for example:
$vsql -h 10.10.51.180 -c "SELECT node_name FROM sessions"
----------------v_ipvs_node01
v_ipvs_node02
v_ipvs_node03
(3 rows)
node_name
Wed Nov
The check.txt is a file located in the /etc/keepalived/ directory, and it gets updated when you
submit changes to the kernel using sysctl -p, described in Disable the Address Resolution
Protocol (ARP). For example, the spread.pl script (see Configuring the Directors), writes to the
check.txt file, which is then modified to include only the remaining nodes in the event of a node
failure. Thus, the virtual server knows to stop sending vsql requests to the failed node.
You can also look for messages by issuing the following command at a shell prompt:
# tail -f /var/log/messages
Nov 3 09:21:00 p6 Keepalived: Starting Keepalived v1.1.17 (05/17,2010)Nov
Keepalived: Starting Healthcheck child process, pid=32468
3 09:21:00 p6
Administrator's Guide
Managing the Database
Administrator's Guide
Managing the Database
2. Install ipvsadm:
[root@node01]# yum install ipvsadmLoading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for ipvsadm to pack into transaction set.
ipvsadm-1.24-10.x86_64.rp 100% |=========================| 6.6 kB
00:00
---> Package ipvsadm.x86_64 0:1.24-10 set to be updated
--> Running transaction check
Dependencies Resolved
=============================================================================
Package
Arch
Version
Repository
Size
=============================================================================
Installing:
ipvsadm
x86_64
1.24-10
base
32 k
Transaction Summary
=============================================================================
Install
1 Package(s)
Update
0 Package(s)
Remove
0 Package(s)
Total download size: 32 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): ipvsadm-1.24-10.x8 100% |=========================| 32 kB
00:00
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: ipvsadm
######################### [1/1]
Installed: ipvsadm.x86_64 0:1.24-10
Complete!
3. Run ipvsadm:
Administrator's Guide
Managing the Database
See Also
l
Resolution
Try to telnet to the VIP and port:
# telnet 10.10.51.180 5433
Administrator's Guide
Managing the Database
Upon startup:
Subject: [node01] VRRP Instance VI_1 - Entering MASTER state=> VRRP Instance is now
owning VRRP VIPs <=
Errors and informational messages from the keepalived daemon are also written to the
/var/log/messages files.
l
Type ip addr list and see the configured VIP addresses for eth0. For example:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback
00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 10.10.51.180/32 brd 127.255.255.255 scope global lo:0
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
Administrator's Guide
Managing the Database
Check iptables and notice the PREROUTING rule on the BACKUP (slave) director. Even
though ipvsadm has a complete list of real servers to manage, it does not route anything as the
prerouting rule redirects packets to the loopback interface.
# /sbin/iptables -t nat -n -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- 0.0.0.0/0 10.10.51.180
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Note: The master and the 3rd node do not have the REDIRECT entry when you run the
same command on those nodes. Only the slave does.
On some kernels, the nat tables does not show by default without the -t parameter, and n is used to avoid long DNS lookups. See the iptables(8) - Linux man page for details.
During failover, it is normal to expect delay in new connection establishment until the slave node
takes control. The delay could be several minutes depending on the load on the cluster. If you
cannot connect to the database, try to telnet to the VIP and port:
# telnet 10.10.51.180 5433
Administrator's Guide
Managing the Database
Managing Nodes
HP Vertica provides the ability to add, remove, and replace nodes on a live cluster that is actively
processing queries. This ability lets you scale the database without interrupting users.
Perform maintenance
Upgrade hardware
Note: Before taking a node down make sure to back up your database. See Backing Up and
Restoring the Database.
1. Check the K-safety level of your cluster. In vsql enter the following query:
SELECT current_fault_tolerance FROM system;
Important: HP Vertica does not recommend a K-safety level of 0. In this case, you should
backup the database before shutting down a node. See Lowering the K-Safety Level to
Allow for Node Removal for more information.
2. Run Administration Tools, select Advanced Menu, and click OK.
3. Select Stop Vertica on Host and click OK.
4. Choose the host that you want to stop and click OK.
5. Return to the Main Menu, select View Database Cluster State, and click OK. The host you
previously stopped now appears DOWN, as shown.
Administrator's Guide
Managing the Database
You can now perform maintenance. See Restart HP Vertica on a Node for details about
restarting HP Vertica on a node.
Node Types
HP Vertica supports the following node types:
Administrator's Guide
Managing the Database
Type
Description
PERMANENT
EPHEMERAL
EXECUTE
STANDBY
Administrator's Guide
Managing the Database
Administrator's Guide
Managing the Database
Fault Groups
Fault groups let you configure HP Vertica for your physical cluster layout to minimize the risk of
correlated failures inherent in your environment, usually caused by shared resources. HP Vertica
Administrator's Guide
Managing the Database
automatically creates fault groups around control nodes (servers that run spread) in large cluster
arrangements, placing nodes that share a control node in the same fault group.
Consider defining your own fault groups specific to your cluster's physical layout if you want to:
l
Reduce the risk of correlated failures. For example, by defining your rack layout, HP Vertica
could tolerate a rack failure.
HP Vertica supports complex, hierarchical fault groups of different shapes and sizes and provides a
fault group script (DDL generator), SQL statements, system tables, and other monitoring tools.
See High Availability With Fault Groups in the Concepts Guide for an overview of fault groups with
a cluster topology example.
Administrator's Guide
Managing the Database
virtual machine hosts. The objects that go into a parent fault group are delimited by spaces; for
example:
<parent> = <child_1> <child_2> <child_n...>
After the first row of parent fault groups, the order in which you write the group descriptions does not
matter. All fault groups that you define in this file must refer back to a parent fault group, either
directly or by being the child of a fault group that is the child of a parent fault group.
For example, to place B into A and both C and D into B, the format is as follows:
A = B
B = C D
The fault_group_ddl_generator.py script returns output like the following. Consider piping the
output to a file so you can edit and reuse the DDL statements later, such as if you want to add
nodes to an existing fault group.
ALTER DATABASE vmartd DROP ALL FAULT GROUP;
CREATE FAULT GROUP rack1;
ALTER FAULT GROUP rack1 ADD NODE node1;
ALTER FAULT GROUP rack1 ADD NODE node2;
ALTER FAULT GROUP rack1 ADD NODE node3;
ALTER FAULT GROUP rack1 ADD NODE node4;
CREATE FAULT GROUP rack2;
ALTER FAULT GROUP rack2 ADD NODE node5;
ALTER FAULT GROUP rack2 ADD NODE node6;
ALTER FAULT GROUP rack2 ADD NODE node7;
ALTER FAULT GROUP rack2 ADD NODE node8;
CREATE FAULT GROUP rack3;
ALTER FAULT GROUP rack3 ADD NODE node9;
CREATE FAULT GROUP vm3_1;
ALTER FAULT GROUP vm3_1 ADD NODE node10;
ALTER FAULT GROUP vm3_1 ADD NODE node11;
Administrator's Guide
Managing the Database
NODE node12;
FAULT GROUP vm3_1;
NODE node13;
NODE node14;
FAULT GROUP vm4_1;
NODE node15;
NODE node16;
FAULT GROUP vm4_2;
Defining fault groups requires careful and thorough network planning. You must have a solid
understanding of your network topology.
Administrator's Guide
Managing the Database
1. As the database administrator, or a user with sudo privileges, log in to one of the target hosts in
the cluster.
2. Run the fault_group_ddl_generator.py. script and include the following arguments:
n
[Optional] The file name of the <filename>.sql script into which you want to write the
DDLstatements.
Example:
$python /opt/vertica/scripts/fault_group_ddl_generator.py
vmartdb faultGroupDescription.txt > faultgroupddl.sql
The above command writes SQL statements to the file called faultgroupddl.sql.
3. Run vsql and log in to the target database as the database administrator (dbadmin by default).
4. Run the .sql output script you created in Step 2; for example:
=> \i faultgroupddl.sql
5. If you do not have large cluster enabled (fewer than 120 nodes), proceed to the next step;
otherwise, you must realign the control nodes by calling the following function:
=> SELECT realign_control_nodes();
Administrator's Guide
Managing the Database
See Also
For syntax on each of the fault group statements and functions, see the following topics in the SQL
Reference Manual:
l
CREATEFAULTGROUP
ALTERFAULTGROUP
DROPFAULTGROUP
ALTERDATABASE
Remove one or more nodes from a fault group (in which case the node is left without a parent
fault group)
Administrator's Guide
Managing the Database
1. As the database administrator or a user with sudo privileges, log in to any host in the cluster.
2. Run the fault groups python script and supply the following:
n
[Optional] The file name of the <filename>.sql script you want to create
Example:
$/opt/vertica/scripts/fault_group_ddl_generator.py
vmartdb ModifiedFaultGroupTemp.txt > faultgroupddl.sql
The above command writes the SQL statements for to the file name, faultgroupddl.sql.
3. Run vsql and log in to the specified database as the database administrator (default dbadmin).
4. Run the <filename>.sql script you created in the step 2; for example:
=> \i faultgroupddl.sql
5. If you do not have large cluster enabled, skip this step; otherwise, you must realign the control
nodes by calling the following function:
=> SELECT realign_control_nodes();
6. Restart the spread process to write changes to the spread configuration file:
=> SELECT reload_spread(true);
See Also
See the following topics in the SQL Reference Manual:
Administrator's Guide
Managing the Database
ALTERFAULTGROUP
ALTERDATABASE
DROPFAULTGROUP
Administrator's Guide
Managing the Database
Administrator's Guide
Managing the Database
While monitoring cluster details on the MCinterface, if information in the node details box is hard to
read, increase the zoom level.
Large Cluster
To support scaling of existing clusters into large clusters and improve control message
performance, HP Vertica delegates control message responsibilities to a subset of HP Vertica
nodes, called control nodes. Control nodes communicate with each other. Other cluster nodes are
assigned to a control node, which they use for control message communications.
Administrator's Guide
Managing the Database
Do this
database
existing databases
Administrator's Guide
Managing the Database
To asses how many cluster nodes should be control nodes, use the square root of the total number
of nodes expected to be in the database cluster to help satisfy both data K-Safety and rack fault
tolerance for the cluster. Depending on the result, you might need to adjust the number of control
nodes to account for your physical hardware/rack count. For example, if you have 121 nodes (with a
result of 11), and your nodes will be distributed across 8 racks, you might want to increase the
number of control nodes to 16 so you have two control nodes per rack.
Administrator's Guide
Managing the Database
Note: The number of hosts you include in the --hosts argument determines a large cluster
layout, not the number of nodes you later include in the database. If you specify 120 or more
hosts in the --hosts list, but you do not specifically enable large cluster by providing the -large-cluster argument, HP Vertica automatically enables large cluster and configures
control nodes for you.
To help control nodes and the nodes assigned to them be configured for the highest possible fault
tolerance, you must specify hosts in the --hosts host_list in a specific order. For example, if
you have four sets of hosts on four racks, the first four nodes in the --hosts host_list must be
one host from each rack in order to have one control node per rack. Then the list must consist of
four hosts from each rack in line with the first four hosts. You'll continue to use this pattern of host
listing for all targeted hosts. See Sample rack-based cluster hosts topology below for examples.
Tip: If you pass the --large-cluster argument a DEFAULT value instead of an <integer>
value, HP Vertica calculates a number of control nodes based on the total number of nodes
specified in the --hosts host_list argument. If you want a specific number of control nodes
on the cluster, you must use the <integer> value.
For more information, see the following topics:
l
Administrator's Guide
Managing the Database
In this scenario, if control node Host-1_1 or Rack-1 fails, the nodes in Rack-1 can communicate
with control node Host-2_1, and the database stays up.
In the following install_vertica script fragment, note the order of the hosts in the --hosts list
argument. The final arguments specifically enable large cluster and provide the number of control
nodes (3):
... install_vertica --hosts Host-1-1,Host-1-2,Host-1-3,Host-2-1,Host-2-2,Host-2-3,Host-3-
Administrator's Guide
Managing the Database
1,Host-3-2,Host-3-3,
Host-4-1,Host-4-2,Host-4-3,Host-5-1,Host-5-2,Host-5-3 -rpm <vertica-package-name> <other
required options>
--large-cluster 3
After the installation process completes, use the Administration Tools to create a database. This
operation generate an HP Vertica cluster environment with three control nodes and their respective
associated hosts that reside on the same racks as the control node.
Administrator's Guide
Managing the Database
4. Call the RELOAD_SPREAD(true) function to save changes to the spread configuration file:
=> SELECT RELOAD_SPREAD(true);
5. After the RELOAD_SPREAD() operation finishes, log back in to the Administration Tools, and
restart the database.
6. Call the REBALANCE_CLUSTER() function to distribute data across the cluster:
=> SELECT REBALANCE_CLUSTER();
Important: You must run REBALANCE_CLUSTER() for fault tolerance to be realized. See
also Rebalancing Large Clusters.
For more details about the functions used in this procedure, see Cluster Management Functions in
the SQL Reference Manual.
Administrator's Guide
Managing the Database
Refreshes projections
When to Rebalance
Rebalancing is useful (or necessary) after you:
l
Mark one or more nodes as ephemeral in preparation of removing them from the cluster
Add one or more nodes to the cluster so that HP Vertica can populate the empty nodes with data
Remove one or more nodes from the cluster so that HP Vertica can redistribute the data among
the remaining nodes
Change the scaling factor of an elastic cluster, which determines the number of storage
containers used to store a projection across the database
Set the control node size or realign control nodes on a large cluster layout
Specify more than 120 nodes in your initial HP Vertica cluster configuration
Administrator's Guide
Managing the Database
See Also
l
Administrator's Guide
Managing the Database
Tip: The CRITICAL_HOSTS view is especially useful for large cluster arrangements. For
non-large clusters, query the CRITICAL_NODES table.
Administrator's Guide
Managing the Database
NODES()and RELOAD_SPREAD(true) functions, which notify nodes of the changes and realign fault
groups. Calling RELOAD_SPREAD(true) connects an existing cluster node to a newly-assigned
control node.
On the other hand, if you run REALIGN_CONTROL_NODES() multiple times in a row, the layout does
not change beyond the initial setup, so you don't need to restart the database. But if you add or drop
a node and then run REALIGN_CONTROL_NODES(), the function call could change many node
assignments.
Here's what happens with control node assignments when you add or drop nodes, whether those
nodes are control nodes or non-control nodes:
l
If you add a cluster nodeHP Vertica assigns a control node to the newly-added node based
on the current cluster configuration. If the new node joins a fault group, it is assigned to a control
node from that fault group and requires a database restart to reconnect to that control node. See
Fault Groups for more information.
If you drop a non-control nodeHP Vertica quietly drops the cluster node. This operation
could change the cluster and spread layout, so you must call REBALANCE_CLUSTER() after you
drop a node.
If you drop a control nodeAll nodes assigned to the control node go down. In large cluster
implementations, however, the database remains up because the down nodes are not buddies
with other cluster nodes.
Dropping a control node results in (n-1) control nodes. You must call REALIGN_CONTROL_NODES
() to reset the cluster so it has n control nodes, which might or might not be the same number as
before you dropped the control node. Remaining nodes are assigned new control nodes. In this
operation, HP Vertica makes control node assignments based on the cluster layout. When it
makes the new assignments, it respects user-defined fault groups, if any, which you can view
by querying the V_CATALOG.CLUSTER_LAYOUT system table, a view that also lets you see the
proposed new layout for nodes in the cluster. If you want to influence the layout of control nodes
in the cluster, you should define fault groups.
For more information, see Defining and Realigning Control Nodes on an Existing Cluster and
Rebalancing Large Clusters.
Administrator's Guide
Managing the Database
Standby node availability is especially important for control nodes. If you are swapping a node
that's a control node, all nodes assigned to the control node's host grouping will need to be taken
offline while you swap in the standby node. For details on node replacement, see Replacing Nodes.
Elastic Cluster
You can scale your cluster up or down to meet the needs of your database. The most common case
is to add nodes to your database cluster to accommodate more data and provide better query
performance. However, you can scale down your cluster if you find that it is overprovisioned or if
you need to divert hardware for other uses.
You scale your cluster by adding or removing nodes. Nodes can be added or removed without
having to shut down or restart the database. After adding a node or before removing a node, HP
Vertica begins a rebalancing process that moves data around the cluster to populate the new nodes
or move data off of nodes about to be removed from the database. During this process data may
Administrator's Guide
Managing the Database
also be exchanged between nodes that are not being added or removed to maintain robust
intelligent K-safety. If HP Vertica determines that the data cannot be rebalanced in a single iteration
due to a lack of disk space, then the rebalance is done in multiple iterations.
To help make data rebalancing due to cluster scaling more efficient, HP Vertica locally segments
data storage on each node so it can be easily moved to other nodes in the cluster. When a new
node is added to the cluster, existing nodes in the cluster give up some of their data segments to
populate the new node and exchange segments to keep the number of nodes that any one node
depends upon to a minimum. This strategy keeps to a minimum the number of nodes that may
become critical when a node fails (see Critical Nodes/K-safety). When a node is being removed
from the cluster, all of its storage containers are moved to other nodes in the cluster (which also
relocates data segments to minimize nodes that may become critical when a node fails). This
method of breaking data into portable segments is referred to as elastic cluster, since it makes
enlarging or shrinking the cluster easier.
The alternative to elastic cluster is to resegment all of the data in the projection and redistribute it to
all of the nodes in the database evenly any time a node is added or removed. This method requires
more processing and more disk space, since it requires all of the data in all projections to
essentially be dumped and reloaded.
Administrator's Guide
Managing the Database
Administrator's Guide
Managing the Database
SET
(1 row)
=> SELECT scaling_factor FROM ELASTIC_CLUSTER;
scaling_factor
--------------6
(1 row)
Administrator's Guide
Managing the Database
containers that are easily moved. Having the data segmented in this way dramatically speeds up
the process of adding or removing nodes, since the data is always in a state that can be quickly
relocated to another node. The rebalancing process that HP Vertica performs after adding or
removing a node just has to decide which storage containers to relocate, instead of first having to
first break the data into storage containers.
Local data segmentation increases the number of storage containers stored on each node. This is
not an issue unless a table contains many partitions. For example, if the table is partitioned by day
and contains one or more years. If local data segmentation is enabled, then each of these table
partitions is broken into multiple local storage segments, which potentially results in a huge number
of files which can lead to ROS "pushback." Consider your table partitions and the effect enabling
local data segmentation may have before enabling the feature.
Administrator's Guide
Managing the Database
Note: You should always perform a database backup before and after performing any of the
operations discussed in this topic. You need to back up before changing any elastic cluster or
local segmentation settings to guard against a hardware failure causing the rebalance process
to leave the database in an unusable state. You should perform a full backup of the database
after the rebalance procedure to avoid having to rebalance the database again if you need to
restore from a backup.
Local segmentation can result in an excessive number of storage containers with tables that have
hundreds of partitions, or in clusters with a non-power-of-two number of nodes. If your database has
these two features, take care when enabling local segmentation.
Enable elastic cluster now, and rebalance the database. This may take a significant amount of
time to complete,. and make consume up to 50% of the free disk space on the nodes in the
database, since all of the segmented projections are re-written. However, afterwards, adding
and removing nodes will take less time.
Wait until you need to resize the cluster, then enable elastic cluster just before adding or
removing nodes. Changing the setting does not make the resizing of the cluster any faster, but
later resize operations will be faster.
Which method you choose depends on your specific circumstances. If you might resize your
database on short notice (for example, you may need to load a very large amount of data at once),
you can choose to schedule the downtime needed to enable elastic cluster and rebalance the
Administrator's Guide
Managing the Database
database to enable elastic cluster sooner, so the actual add or remove node process will occur
faster.
If you choose to enable elastic cluster for your database, you should consider whether you want to
enable local data segmentation at the same time. If you choose to enable local data segmentation
at a later time, you will need to rebalance the database again, which is a lengthy process.
Note: If multiple rebalance methods were used for a single table (for example, the table has
unsegmented and segmented projections), the table may appear multiple times - once for
each rebalance method.
REBALANCE_PROJECTION_STATUS can be used to gain more insight into the details for a
particular projection that is being rebalanced. It provides the same type of information as above,
but in terms of a projection instead of a table.
Administrator's Guide
Managing the Database
Adding Nodes
There are many reasons for adding one or more nodes to an installation of HP Vertica:
l
Increase system performance. Add additional nodes due to a high query load or load latency or
increase disk space without adding storage locations to existing nodes.
Note: The database response time depends on factors such as type and size of the application
query, database design, data size and data types stored, available computational power, and
network bandwidth. Adding nodes to a database cluster does not necessarily improve the
system response time for every query, especially if the response time is already short, e.g.,
less then 10 seconds, or the response time is not hardware bound.
Make the database K-safe (K-safety=1) or increase K-safety to 2. See Failure Recovery for
details.
Swap a node for maintenance. Use a spare machine to temporarily take over the activities of
an existing node that needs maintenance. The node that requires maintenance is known ahead
of time so that when it is temporarily removed from service, the cluster is not vulnerable to
additional node failures.
Important: : If you installed HP Vertica on a single node without specifying the IP address or
hostname (or you used localhost), you cannot expand the cluster. You must reinstall HP
Vertica and specify an IP address or hostname that is not localhost/127.0.0.1.
Adding nodes consists of the following general tasks:
1. Back up the database.
HP strongly recommends that you back up the database before you perform this significant
operation because it entails creating new projections, refreshing them, and then deleting the old
projections. See Backing Up and Restoring the Database for more information.
The process of migrating the projection design to include the additional nodes could take a
while; however during this time, all user activity on the database can proceed normally, using
the old projections.
2. Configure the hosts you want to add to the cluster.
Administrator's Guide
Managing the Database
See Before you Install HP Vertica in the Installation Guide. You will also need to edit the hosts
configuration file on all of the existing nodes in the cluster to ensure they can resolve the new
host.
3. Add one or more hosts to the cluster.
4. Add the hosts you added to the cluster (in step 3) to the database.
Note: When you add a "host" to the database, it becomes a "node." You can add nodes to
your database using either the Administration Tools or the Management Console (See
Monitoring HP Vertica Using Management Console.)
After you add one or more nodes to the database, HP Vertica automatically distributes updated
configuration files to the rest of the nodes in the cluster and starts the process of rebalancing data in
the cluster. See Rebalancing Data Across Nodes for details.
If you installed HP Vertica on a single node without specifying the IP address or hostname (you
used localhost), it is not possible to expand the cluster. You must reinstall HP Vertica and
specify an IP address or hostname.
If your database has more than one node already, you can add a node without stopping the
server. However, if you are adding a node to a single-node installation, then you must shut down
both the database and spread. If you do not, the system returns an error like the following:
$sudo /opt/vertica/sbin/update_vertica --add-hosts node05 --rpm vertica_7.1.x.x86_
64.RHEL5.rpm
Vertica 7.0.x Installation Tool
Starting installation tasks...
Getting system information for cluster (this may take a while)....
Spread is running on ['node01']. HP Vertica and spread must be stopped before adding
nodes to a 1 node cluster.
Use the admin tools to stop the database, if running, then use the following command to
Administrator's Guide
Managing the Database
stop spread:
/etc/init.d/spread stop (as root or with sudo)
Installation completed with errors.
Installation failed.
Note: See Installing with the Script for the full list of parameters. You must also provide the
same options you used when originally installing the cluster.
The update_vertica script uses all the same options as install_vertica and:
l
Performs post-installation checks, including RPM version and N-way network connectivity
checks.
Important Tips:
l
A host can be specified by the hostname or IP address of the system you are adding to the
cluster. However, internally HP Vertica stores all host addresses as IP addresses.
Do not use include spaces in the hostname/IP address list provided with --add-hosts if you
specified more than one host.
If a package is specified with --rpm/--deb, and that package is newer than the one currently
installed on the existing cluster, then, HP Vertica first installs the new package on the existing
cluster hosts before the newly-added hosts.
Administrator's Guide
Managing the Database
Use the same command line parameters for the database administrator username, password,
and directory path you used when you installed the cluster originally. Alternatively, you can
create a properties file to save the parameters during install and then re-using it on subsequent
install and update operations. See Installing HP Vertica Silently.
If you are installing using sudo, the database administrator user (dbadmin) must already exist on
the hosts you are adding and must be configured with passwords and home directory paths
identical to the existing hosts. HP Vertica sets up passwordless ssh from existing hosts to the
new hosts, if needed.
If you initially used the --point-to-point option to configure spread to use direct, point-to-point
communication between nodes on the subnet, then use the --point-to-point option whenever
you run install_vertica or update_vertica. Otherwise, your cluster's configuration is
reverted to the default (broadcast), which may impact future databases.
It is possible to have more than 80 nodes by using large cluster mode, which does not install a
spread daemon on each node.
Examples:
--add-hosts host01 --rpm
--add-hosts 192.168.233.101
--add-hosts host02,host03
Administrator's Guide
Managing the Database
You add nodes to a database on MC's Manage page. Click the node you want to act upon, and then
click Add node in the Node List.
When you add a node, the node icon in the cluster view changes color from gray (empty) to green as
the node comes online. Additionally, a task list displays detailed progress of the node addition
process.
Administrator's Guide
Managing the Database
database has a lower load. If you choose to automatically rebalance the database, the script is
still created and saved where you can use it later.
14. Review the summary of the rebalancing process and select Proceed.
15. If you chose to automatically rebalance, the rebalance process runs. If you chose to create a
script, the script is generated and saved. In either case, you are shown a success screen, and
prompted to select OK to end the Add Node process.
Administrator's Guide
Managing the Database
Removing Nodes
Although less common than adding a node, permanently removing a node is useful if the host
system is obsolete or over-provisioned.
Note: You cannot remove nodes if your cluster would not have the minimum number of nodes
required to maintain your database's current K-safety level (3 nodes for a database with a Ksafety level of 1, and 5 nodes for a K-safety level of 2). If you really wish to remove the node or
nodes from the database, you first must reduce the K-safety level of your database.
Removing one or more nodes consists of the following general steps:
1. Back up the database.
HP recommends that you back up the database before performing this significant operation
because it entails creating new projections, deleting old projections, and reloading data.
2. Lower the K-safety of your database if the cluster will not be large enough to support its current
level of K-safety after you remove nodes.
3. Remove the hosts from the database.
4. Remove the nodes from the cluster if they are not used by any other databases.
Administrator's Guide
Managing the Database
2. Enter the command: SELECT MARK_DESIGN_KSAFE(n); where n is the new K-Safety level for
the database (0 if you are reducing the cluster to below 3 nodes, 1 if you are reducing the
cluster to 3 or 4 nodes).
Prerequisites
l
The node must be empty, in other words there should be no projections referring to the node.
Ensure you have followed the steps listed in Removing Nodes to modify your database design.
You cannot drop nodes that are critical for K-safety. See Lowering the K-Safety Level to Allow
for Node Removal.
Administrator's Guide
Managing the Database
3. From the Main Menu, select Advanced Tools Menu, and then select OK.
4. In the Advanced menu, select Cluster Management, and then select OK.
5. In the Cluster Management menu, select Remove Host(s) from Database, and then select
OK.
6. When warned that you must redesign your database and create projections that exclude the
hosts you are going to drop, select Yes.
7. Select the database from which you want to remove the hosts, and then select OK.
A list of all the hosts that are currently being used is displayed.
8. Select the hosts you want to remove from the database, and then select OK.
9. When prompted, select OK to confirm that you want to remove the hosts. HP Vertica begins
the process of rebalancing the database and removing the node or nodes.
10. When informed that the hosts were successfully removed, select OK.
11. If you removed a host from a Large Cluster configuration, open a vsql session and run the
following command:
SELECT realign_control_nodes();
Prerequisites
The host must not be used by any database.
Administrator's Guide
Managing the Database
Note: See Installing with the Script for the full list of parameters.
The update_vertica script uses all the same options as install_vertica and:
l
Important Tips:
l
Do not include spaces in the hostname list provided with --remove-hosts if you specified more
than one host.
If a new RPM is specified with --rpm, then HP Vertica will first install it on the existing cluster
hosts before proceeding.
Use the same command line parameters as those used when you installed the original cluster.
Specifically if you used non-default values for the database administrator username, password,
or directory path, provide the same when you remove hosts; otherwise; the procedure fails.
Consider creating a properties file in which you save the parameters during the installation,
which you can reuse on subsequent install and update operations. See Installing HP Vertica
Silently.
Examples:
--remove-hosts host01
--remove-hosts 192.168.233.101
-R host01
Administrator's Guide
Managing the Database
Replacing Nodes
If you have a K-Safe database, you can replace nodes, as necessary, without bringing the system
down. For example, you might want to replace an existing node if you:
l
Need to repair an existing host system that no longer functions and restore it to the cluster
Want to exchange an existing host system for another more powerful system
Note: HP Vertica does not support replacing a node on a K-safe=0 database. Use the
procedures to add and remove nodes instead.
The process you use to replace a node depends on whether you are replacing the node with:
l
Prerequisites
l
Configure the replacement hosts for HP Vertica. See Before you Install HP Vertica in the
Installation Guide.
Read the Important Tips sections under Adding Hosts to a Cluster and Removing Hosts From
a Cluster.
Ensure that the database administrator user exists on the new host and is configured identically
to the existing hosts. HP Vertica will setup passwordless ssh as needed.
Ensure that directories for Catalog Path, Data Path, and any storage locations are added to the
database when you create it and/or are mounted correctly on the new host and have read and
write access permissions for the database administrator user. Also ensure that there is
sufficient disk space.
Follow the best practice procedure below for introducing the failed hardware back into the cluster
to avoid spurious full-node rebuilds.
Administrator's Guide
Managing the Database
Note: You do not need to initialize RAID recover in runlevel 1; simply validate that it can
recover.
3. Once the hardware is confirmed consistent, only then reboot to runlevel 3 or higher.
At this point, the network activates, and HP Vertica rejoins the cluster and automatically recovers
any missing data. Note that, on a single-node database, if any files that were associated with a
projection have been deleted or corrupted, HP Vertica will delete all files associated with that
projection, which could result in data loss.
The installation script verifies system configuration and that HP Vertica, spread, and the
Administration Tools metadata are installed on the host.
Administrator's Guide
Managing the Database
3. On the new node, create catalog and data directories (unless they both reside in the same toplevel directory, then you just need to create the one directory). These are the same top-level
directories you specified when creating the database.
Note: You can find the directories used for catalog and data storage by querying the V_
MONITOR.DISK_STORAGE system table. You need to create the directories up to the
v_database_node00xx portion of the data and catalog path. For example, if the catalog
storage location is /home/dbadmin/vmart/v_vmart_node0001_catalog/Catalog, you would
need to create the /home/dbadmin/vmart directory to store the catalog.
Note: Do not connect two hosts with the same name and IP address to the same network.
If this occurs, traffic is unlikely to be routed properly.
Administrator's Guide
Managing the Database
5. Use the procedure in Distributing Configuration Files to the New Host to transfer metadata to
the new host.
6. Remove the host from the cluster.
7. Use the Administration Tools to restart HP Vertica on the host. On the Main Menu, select
Restart Vertica on Host, and click OK. See Starting the Database for more information.
Once you have completed this process, the replacement node automatically recovers the data that
was stored in the original node by querying other nodes within the database.
Administrator's Guide
Managing the Database
database, the host is not invited to join the database because its node address does not match
the new address stored in the database catalog. Therefore, it remains in the INITIALIZING
state.
Administrator's Guide
Managing the Database
14. When prompted, click Yes to confirm that you want to replace the host.
15. When prompted that the host was successfully replaced, click OK.
16. In the Main Menu, select View Database Cluster State to verify that all the hosts are running.
You might need to start HP Vertica on the host you just replaced. Use Restart Vertica on
Host.
The node enters a RECOVERING state.
Caution: If you are using a K-Safe database, keep in mind that the recovering node counts as
one node down even though it might not yet contain a complete copy of the data. This means
that if you have a database in which K safety=1, the current fault tolerance for your database is
at a critical level. If you lose one more node, the database shuts down. Be sure that you do not
stop any other nodes.
Administrator's Guide
Managing the Database
For segmented projections, HP Vertica creates new (renamed), segmented projections that are
identical in structure to the existing projections, but which have their data distributed across all
nodes. The rebalance process then refreshes all new projections, sets the Ancient History Mark
(AHM) to the greatest allowable epoch (now), and drops all of the old segmented projections. All
new buddy projections have the same base name so they can be identified as a group.
Note: HP Vertica does not maintain custom projection segmentations defined with a
specific node list. Node rebalancing distributes data across all nodes, regardless of any
custom definitions.
For unsegmented projections, leaves existing projections unmodified, creates new projections
on the new nodes, and refreshes them.
Administrator's Guide
Managing the Database
Permissions
Only the superuser has permissions to rebalance data.
Administrator's Guide
Managing the Database
The Database Designer modifies existing projections to rebalance data across all database
nodes with the K-safety you provided. A script to rebalance data, which you can run manually
at a later time, is also generated and resides in the path you specified; for example
/tmp/extend_catalog_rebalance.sql.
Important: Rebalancing data can take some time, depending on the number of projections
and the amount of data they contain. HP recommends that you allow the process to
complete. If you must cancel the operation, use Ctrl+C.
The terminal window notifies you when the rebalancing operation is complete.
10. Press Enter to return to the Administration Tools.
REBALANCE_CLUSTER()
START_REBALANCE_CLUSTER()
CANCEL_REBALANCE_CLUSTER()
For more information and examples of using these functions, see their entries in the SQL Reference
Manual.
Administrator's Guide
Managing the Database
Note: To distribute configuration files from the command line or via scripts, use the
admintools option distribute_config_files:
admintools -t distribute_config_files
Administrator's Guide
Managing the Database
Administrator's Guide
Managing the Database
Description
DISK_STORAGE
Monitors the amount of disk storage used by the database on each node.
COLUMN_STORAGE
PROJECTION_STORAGE
Monitors the amount of disk storage used by each projection on each node.
Administrator's Guide
Managing the Database
Note: If you are adding disk space to multiple nodes in the cluster, then use the following
procedure for each node, one node at a time.
To add disk space to a node:
1. If you must shut down the hardware to which you are adding disk space, then first shut down
HP Vertica on the host where disk space is being added.
2. Add the new disk to the system as required by the hardware environment. Boot the hardware if
it is was shut down.
3. Partition, format, and mount the new disk, as required by the hardware environment.
4. Create a data directory path on the new volume.
For example:
mkdir p /myNewPath/myDB/host01_data2/
5. If you shut down the hardware, then restart HP Vertica on the host.
6. Open a database connection to HP Vertica and add a storage location to add the new data
directory path. Specify the node in the CREATE LOCATION, otherwise HP Vertica assumes
you are creating the storage location on all nodes.
See Adding Storage Locations in this guide and the CREATE LOCATION statement in the
SQL Reference Manual.
Administrator's Guide
Managing the Database
Note: DATABASE_HOME_DIR is the path, which you can see from the Administration Tools.
See Using the Administration Tools in the Administrator's Guide for details on using the
interface.
To view the path of your database:
1. Run the Administration Tools.
$/opt/vertica/bin/admintools
2. From the Main Menu, select Configuration Menu and click OK.
3. Select View Database and click OK.
4. Select the database you want would like to view and click OK to see the database profile.
See Understanding the Catalog Directory for an explanation of the contents of the catalog directory.
Administrator's Guide
Managing the Database
where nodennnn is the name of the node you are logged into. The name of the catalog directory is
unique for each node, although most of the contents of the catalog directory are identical on each
node.
The following table explains the files and directories that may appear in the catalog directory.
Note: Do not change or delete any of the files in the catalog directory unless asked to do so by
HP Vertica support.
File or Directory
Description
bootstrap-catalog.log
Catalog/
CopyErrorLogs/
DataCollector/
debug_log.conf
Epoch.log
Administrator's Guide
Managing the Database
File or Directory
Description
ErrorReport.txt
Libraries/
Snapshots/
tmp/
UDxLogs/
vertica.conf
vertica.log
vertica.pid
Administrator's Guide
Managing the Database
Rebuilding a Table
When it is necessary to do large-scale disk reclamation operations, consider rebuilding the table by
following sequence of operations:
1. Create a new table.
2. Create projections for the new table.
3. Populate the new table using INSERT ... SELECT to copy the desired table from the old table.
4. Drop the old table and its projections.
5. Use ALTER TABLE ... RENAME to give the new table the name of the old table.
Notes
l
You must have enough disk space to contain the old and new projections at the same time. If
necessary, you can drop some of the old projections before loading the new table. You must,
however, retain at least one superprojection of the old table (or two buddy superprojections to
maintain K-safety) until the new table is loaded. (See Prepare Disk Storage Locations in the
Installation Guide for disk space requirements.)
You can specify different names for the new projections or use the ALTER PROJECTION ...
RENAME command to change the names of the old projections.
The relationship between tables and projections does not depend on object names. Instead, it
depends on object identifiers that are not affected by rename operations. Thus, if you rename a
table, its projections continue to work normally.
Manually purging a table continues to retain history for rows deleted after the Ancient History
Mark. Rebuilding the table results in purging all the history of the table, which means you cannot
do historical queries on any older epoch.
Rather than dropping the old table in Step 4, you might rename it to a different name and use it as
a backup copy. Note, however, that you must have sufficient disk space.
Administrator's Guide
Managing the Database
Administrator's Guide
Administrator's Guide
Moveout
Mergeout
Each of these operations occurs at different intervals across all nodes. The tuple mover runs
independently on each node, ensuring that storage is managed appropriately even in the event of
data skew.
Administrator's Guide
Moveout
Moveout operations move data from memory (WOS) into a new ROS container. A moveout
"flushes" all historical data from the WOS to the ROS.
The following illustration shows the effect of a projection moveout on a single node:
ROS Containers
A ROS (Read Optimized Store) container is a set of rows stored in a particular group of files. ROS
containers are created by operations like Moveout or COPY DIRECT, and can be observed in the
STORAGE_CONTAINERS system table. The ROS container layout can differ across nodes due to data
variance. Segmentation can deliver more rows to one node than another. Two loads could fit in the
WOS on one node and spill on another.
Mergeout
A mergeout is the process of consolidating ROS containers and purging deleted records. Over time,
the number of ROS containers increases to a degree that it becomes necessary to merge some of
them in order to avoid performance degradation. At that point, the Tuple Mover performs an
Administrator's Guide
automatic mergeout, which combines two or more ROS containers into a single container. This
process can be thought of as "defragmenting" the ROS.
Vertica keeps data from different partitions separate on disk.When the Tuple Mover consolidates
ROS containers, it adheres to this policy by not merging ROS containers from different
partitions.When a partition is first created, it is typically the subject of frequent data loads and
requires regular attention from the Tuple Mover.As a partition ages, it commonly transitions to a
read-only workload that requires much less attention.
The Tuple Mover has two different policies for managing these different partition workloads:
l
Active partitions are loaded or modified frequently. The Tuple Mover uses a STRATA mergeout
policy that keeps a collection of ROS container sizes to minimize the number of times any
individual tuple is subjected to mergeout. The ActivePartitionCount parameter identifies how
many partitions are being actively loaded.
Inactive partitions are very infrequently loaded or modified. The Tuple Mover consolidates the
ROS containers to a minimal set while avoiding merging containers whose size exceeds
MaxMrgOutROSSizeMB.
Partitions are not explicitly marked by the user as active or inactive; instead, the Tuple Mover uses
the following algorithm to order the partitions from oldest to newest:
l
If two partitions were created at the same time, but one partition was last updated earlier than
the other partition, it is older.
If two partitions were created and last updated at the same time, the partition with the smaller
key is considered older.
If you perform a manual mergeout using the DO_TM_TASK function, all partitions are consolidated into
the smallest possible number of containers, regardless of the value of the ActivePartitionCount
parameter.
Administrator's Guide
After it performs a mergeout, the Tuple Mover looks for deletion marker containers that hold few
entries. If it finds some, it merges them together into a single larger container. This process helps
lower the overhead of tracking deleted data by freeing resources used by the individual containers.
It does not purge or otherwise affect the deleted datait just consolidates the deletion mark
containers for greater efficiency.
Note: You can see the number and size of the containers holding the deletion marks by
viewing the V_MONITOR.DELETE_VECTORS system table.
Description
ActivePartitionCount
MergeOutInterval
Administrator's Guide
Parameters
Description
MoveOutInterval
MoveOutMaxAgeTime
MoveOutSizePct
The percentage of the WOS that can be filled with data before
the Tuple Mover performs a moveout operation.
Default Value: 0
Example:
ALTERDATABASE mydb SET MoveOutSizePct = 50;
Administrator's Guide
latency in read and write, adjust the PLANNEDCONCURRENCY parameter to keep the storage
subsystem resources below saturation level. In addition, you might need to:
l
The following statement illustrates how to increase the size of the TM resource pool and set the
concurrency settings for the pool:
=> ALTER RESOURCE POOL tm MEMORYSIZE '4G' PLANNEDCONCURRENCY 4 MAXCONCURRENCY 5;
The WOSDATA resource pool settings also indirectly affect the Tuple Mover. In automatic mode,
INSERT and COPY commands use the concurrency setting to determine whether data is small
enough to store in WOS or if it should be written to ROS. Therefore, set this value to be the number
of concurrent loads you expect to perform in your database. The WOSDATA resource pool also
determines how much RAM the WOS can use.
=> ALTER RESOURCE POOL wosdata MAXMEMORYSIZE '4G' PLANNEDCONCURRENCY 3;
See Managing Workloads and Resource Pool Architecture in this guide and ALTER RESOURCE
POOL and Built-in Pools in the SQL Reference Manual.
Loading Data
HP Vertica automatically decides whether the data should be placed in WOS or stored directly in
ROS containers based on the amount of data processed by a COPY or INSERT command. HP
Vertica stores large loads directly to disk and stores smaller loads in memory, which it later moves
to disk.
For low-latency access to data, use small loads. The automatic Tuple Mover settings are the best
option for handling such smaller loads. One exception is for single-node deployments, where a
system failure would cause in-memory data to be lost. In this case, you might want to force all data
loads to go directly to disk.
For high load rates, you might want the Tuple Mover to check for jobs more frequently by changing
the MergeOutInterval and MoveOutInterval configuration parameters. Reduce the
MoveOutInterval if you expect the peak load rate to fill the WOS quickly. Reduce
MergeOutInterval if you anticipate performing many DIRECT loads or inserts.
See COPY and INSERT in the SQL Reference Manual
Administrator's Guide
See ALTER RESOURCE POOL and Built-in Pools in the SQL Reference Manual.
See Also
l
Administrator's Guide
Managing Workloads
HP Vertica provides a sophisticated resource management scheme that allows diverse, concurrent
workloads to run efficiently on the database. For basic operations, the built-in GENERAL pool is
pre-configured based on RAM and machine cores, but you can customized this pool to handle
specific concurrency requirements.
You can also define new resource pools that you configure to limit memory usage, concurrency,
and query priority. You can then optionally restrict each database user to use a specific resource
pool, which control memory resources used by their requests.
User-defined pools are useful if you have competing resource requirements across different
classes of workloads. Example scenarios include:
l
A large batch job takes up all server resources, leaving small jobs that update a web page to
starve, which can degrade user experience.
In this scenario, you can create a resource pool to handle web page requests and ensure users
get resources they need. Another option is to create a limited resource pool for the batch job, so
the job cannot use up all system resources.
A certain application has lower priority than other applications, and you would like to limit the
amount of memory and number of concurrent users for the low-priority application.
In this scenario, you could create a resource pool with an upper limit on the query's memory and
associate the pool with users of the low-priority application.
You can also use resource pools to manage resources assigned to running queries. You can assign
a run-time priority to a resource pool, as well as a threshold to assign different priorities to queries
with different durations. See Managing Resources At Query Run Time for more information.
For detailed syntax of creating and managing resource pools see the following topics in the SQL
Reference Manual:
Statements
l
ALTER USER associates a user with the RESOURCE POOL and MEMORYCAP parameters.
Administrator's Guide
CREATE USER adds a name to the list of authorized database users and specifies that user's
RESOURCE POOL and MEMORYCAP parameters.
SET SESSION MEMORYCAP sets the limit on amount of memory that any request issued by
the session can consume.
SET SESSION RESOURCE POOL associates a user session with specified resource pool.
System Tables
l
RESOURCE_POOLS displays information about the parameters the resource pool was
configured with.
Administrator's Guide
See Also
l
Analyzing Workloads
If the memory required by the query alone would exceed the machine's physical memory,
the query is rejected - it cannot possibly run. Outside of significantly under-provisioned
nodes, this case is very unlikely.
If the resource requirements are not currently available, the query is queued. The query will
remain on the queue until either sufficient resources are freed up and the query runs or the
Administrator's Guide
3. The query starts running when all participating nodes allow it to run.
Note: Once the query is running, the Resource Manager further manages resource allocation
using RUNTIMEPRIORITY and RUNTIMEPRIORITYTHRESHOLD parameters for the resource pool.
See Managing Resources At Query Run Time for more information.
Apportioning resources for a specific query and the maximum number of queries allowed to run
depends on the resource pool configuration. See Resource Pool Architecture.
On each node, no resources are reserved or held while the query is in the queue. However, multinode queries queued on some nodes will hold resources on the other nodes. HP Vertica makes
every effort to avoid deadlocks in this situation.
Administrator's Guide
Administrator's Guide
Administrator's Guide
exist as a user-defined pool or the GENERAL pool. When using CASCADE TO, you cannot create a
resource pool loop.
This example demonstrates a situation where the administrator wants user1's queries to start on
the user resource pool, but cascade to the userOverflow pool if the queries are too long.
=> CREATE RESOURCEPOOL userOverflow RUNTIMECAP '5 minutes';
=> CREATERESOURCEPOOL user RUNTIMECAP '1 minutes' CASCADETOuserOverflow;
=> CREATEUSER "user1"RESOURCEPOOL user;
In this scenario, user1 cannot start his or her queries on the userOverflow resource pool, but
because grant privileges are not considered for secondary pools, user1's queries can cascade to
the userOverflow pool if they exceed the user pool RUNTIMECAP. Using the secondary pool frees
up space in the primary pool so short queries can run.
This example shows a situation where the administrator wants long-running queries to stay queued
on a secondary pool.
=> CREATERESOURCEPOOL rp2 PRIORITYHOLD;
=> CREATERESOURCEPOOL rp1 RUNTIMECAP '2 minutes' CASCADETO rp2;
=> SETSESSIONRESOURCE_POOL = rp1;
In this scenario, queries that run on rp1 for more than 2 minutes will queue on rp2 until
QUEUETIMEOUTis reached, at which point the queries will be rejected.
Parameter Considerations
The secondary resource pool's CPUAFFINITYSETand CPUAFFINITYMODE will be applied to
the query when it enters the pool.
Administrator's Guide
The query will adopt the secondary pool's RUNTIMEPRIORITYat different times, depending on
the following circumstances:
l
If the RUNTIMEPRIORITYTHRESHOLD timer was not started when the query was running in
the primary pool, the query will adopt the secondary resource pools' RUNTIMEPRIOIRTY when
it cascades. This happens either when the RUNTIMEPRIORITYTHRESHOLD is not set for the
primary pool or the RUNTIMEPRIORITY is set to HIGH for the primary pool.
If the RUNTIMEPRIORITYTHRESHOLD was reached in the primary pool, the query will adopt
the secondary resource pools' RUNTIMEPRIOIRTY when it cascades.
If the RUNTIMEPRIORITYTHRESHOLD was not reached in the primary pool and the
secondary pool has no threshold, the query will adopt the new pool's RUNTIMEPRIOIRTY
when it cascades.
If the RUNTIMEPRIORITYTHRESHOLD was not reached in the primary pool and the
secondary pool has a threshold set:
In this case, the query runs for 5 seconds on the primary pool and then cascades to the
secondary pool. After another 3 seconds, 8 seconds total, the query adopts the
RUNTIMEPRIORITY of the secondary pool.
n
Administrator's Guide
In this case, the query runs for 5 seconds on the primary pool and then cascades to the
secondary pool. After another 7 seconds, 12 seconds total, the query adopts the
RUNTIMEPRIORITY of the secondary pool.
Privileges
You must be a superuser to use the CASCADE TO parameter.
Resource pool GRANT USAGE privileges are not considered for secondary pools.
Notes
When a query is moved to another pool, the initial pool regains the memory previously used by the
query.
See Also
For examples, see ALTER RESOURCE POOL and CREATE RESOURCE POOL.
Examples
This example shows how to find the various resource pools defined in the system.
=> SELECT name, memorysize, maxmemorysize FROM V_CATALOG.RESOURCE_POOLS;
name
| memorysize | maxmemorysize
Administrator's Guide
----------+------------+--------------general |
| Special: 95%
sysquery | 64M
|
sysdata | 100M
| 10%
wosdata | 0%
| 25%
tm
| 200M
|
refresh | 0%
|
recovery | 0%
|
dbd
| 0%
|
jvm
| 0%
| 10%
(9 rows)
Administrator's Guide
Expanded display is on
=> SELECT pool_name, memory_size_kb, memory_size_actual_kb, memory_inuse_kb,
general_memory_borrowed_kb,running_query_count
FROM V_MONITOR.RESOURCE_POOL_STATUS where node_name ilike '%node0001';
-[RECORD 1 ]--------------+--------pool_name
| general
memory_size_kb
| 2983177
memory_size_actual_kb
| 2983177
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 2 ]--------------+--------pool_name
| sysquery
memory_size_kb
| 65536
memory_size_actual_kb
| 65536
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 3 ]--------------+--------pool_name
| sysdata
memory_size_kb
| 102400
memory_size_actual_kb
| 102400
memory_inuse_kb
| 4096
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 4 ]--------------+--------pool_name
| wosdata
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 5 ]--------------+--------pool_name
| tm
memory_size_kb
| 204800
memory_size_actual_kb
| 204800
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 6 ]--------------+--------pool_name
| refresh
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 7 ]--------------+--------pool_name
| recovery
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 8 ]--------------+--------pool_name
| dbd
memory_size_kb
| 0
Administrator's Guide
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 9 ]--------------+--------pool_name
| jvm
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
You can determine how long a query waits in the queue before it can run. To do so, you obtain the
difference between the acquisition_timestamp and the queue_entry_timestamp using a query
as this example shows:
=> SELECT pool_name, queue_entry_timestamp, acquisition_timestamp,
(acquisition_timestamp-queue_entry_timestamp) AS 'queue wait'
FROM V_MONITOR.RESOURCE_ACQUISITIONS WHERE node_name ILIKE '%node0001';
-[RECORD 1 ]---------+-----------------------------pool_name
| sysquery
Administrator's Guide
See Also
l
See the SQL Reference Manual for detailed descriptions of the monitoring tables.
See Monitoring and Configuring Resource Pools in Management Console for descriptions of
other ways to monitor resource usage.
User Profiles
User profiles are attributes associated with a user that control that user's access to several system
resources. These resources include:
l
Maximum amount of temporary file storage a user's session can use (TEMPSPACECAP)
You can set these attributes with the CREATE USER statement and modify the attributes later
with ALTER USER.
Two strategies limit a user's access to resources: Setting attributes on the user directly to control
resource use, or assigning the user to a resource pool. The first method lets you fine tune individual
users, while the second makes it easier to group many users together and set their collective
resource usage.
Administrator's Guide
The following examples illustrate how to set a user's resource pool attributes. For additional
examples, see the scenarios described in Using User-Defined Pools and User-Profiles for
Workload Management.
Example
Set the user's RESOURCE POOL attribute to assign the user to a resource pool. To create a user
named user1 who has access to the resource pool my_pool, use the command:
=> CREATE USER user1 RESOURCE POOL my_pool;
To limit the amount of memory for a user without designating a pool, set the user's MEMORYCAP to
either a particular unit or a percentage of the total memory available. For example, to create a user
named user2 whose sessions are limited to using 200 MBs memory each, use the command:
=> CREATE USER user2 MEMORYCAP '200M';
To limit the time a user's queries are allowed to run, set the RUNTIMECAP attribute. To prevent
queries for user2 from running more than five minutes, use this command:
=> ALTER USER user2 RUNTIMECAP '5 minutes';
To limit the amount of temporary disk space that the user's sessions can use, set the
TEMPSPACECAP to either a particular size or a percentage of temporary disk space available. For
example, the next statement creates user3, and limits her to using 1 GB of temporary space:
=> CREATE USER user3 TEMPSPACECAP '1G';
You can combine different attributes into a single command. For example, to limit the MEMORYCAP
and RUNTIMECAP for user3, include both attributes in an ALTER USER statement:
=> ALTER USER user3 MEMORYCAP '750M' RUNTIMECAP '10 minutes';
ALTER USER
=> \x
Expanded display is on.
=> SELECT * FROM USERS;
-[RECORD 1 ]-----+-----------------user_id
| 45035996273704962
user_name
| release
is_super_user
| t
resource_pool
| general
memory_cap_kb
| unlimited
temp_space_cap_kb | unlimited
run_time_cap
| unlimited
-[RECORD 2 ]-----+------------------
Administrator's Guide
user_id
| 45035996273964824
user_name
| user1
is_super_user
| f
resource_pool
| my_pool
memory_cap_kb
| unlimited
temp_space_cap_kb | unlimited
run_time_cap
| unlimited
-[RECORD 3 ]-----+-----------------user_id
| 45035996273964832
user_name
| user2
is_super_user
| f
resource_pool
| general
memory_cap_kb
| 204800
temp_space_cap_kb | unlimited
run_time_cap
| 00:05
-[RECORD 4 ]-----+-----------------user_id
| 45035996273970230
user_name
| user3
is_super_user
| f
resource_pool
| general
memory_cap_kb
| 768000
temp_space_cap_kb | 1048576
run_time_cap
| 00:10
See Also
l
ALTER USER
CREATE USER
For the GENERAL pool, the target amount of memory for the query is calculated using the
Queueing Threshold of the GENERAL pool / PLANNEDCONCURRENCY. Queueing threshold
of the general pool is calculated as MAXMEMORYSIZE * 95%.
If MEMORYSIZE is set to 0 and MAXMEMORYSIZE is not set, the target amount of memory
for the query is calculated using the Queueing Threshold of the GENERAL pool /
Administrator's Guide
If MEMORYSIZEis set to 0 and MAXMEMORYSIZE is set, the target amount of memory for
the query is calculated using the Queueing Threshold of the pool /
PLANNEDCONCURRENCY. Queuing threshold of the pool is based off the
MAXMEMORYSIZE of the pool.
If MEMORYSIZE is set for the pool other than the GENERAL pool, the target memory is set to
Queuing Threshold of the pool/ PLANNEDCONCURRENCY of the pool. Queuing threshold of
the pool is based off MEMORYSIZE.
See Also
l
User Profiles
RESOURCE_POOL_STATUS
Setting Run-Time Priority for the Resource Pool--Use resource pool parameters to set the run
time priority for queries running within the resource pool.
Changing Run-Time Priority of a Running Query--Manually change the run time priority of a
running query.
Administrator's Guide
assigned greater runtime resources than those in resource pools with MEDIUM or LOW runtime
priorities.
Administrator's Guide
Parameter
Description
RUNTIMEPRIORITY
HIGH
MEDIUM
LOW
Queries with a HIGH run-time priority receive more CPU and I/O
resources than those with a MEDIUM or LOW run-time priority.
Note: If you are upgrading from an HP Vertica version prior to
6.0, RUNTIMEPRIORITY is set to LOW. If upgrading from
version 6.0 or higher, RUNTIMEPRIORITY retains its value.
RUNTIMEPRIORITYTHRESHOLD
See Also
l
Administrator's Guide
Using CHANGE_RUNTIME_PRIORITY
The CHANGE_RUNTIME_PRIORITY function allows you to change the priority of a running
query. Introduced in a previous release, this function required you to specify both the transaction ID
and the statement ID of the query whose priority you wanted to change. With the introduction of
CHANGE_CURRENT_STATEMENT_RUNTIME_PRIORITY, you no longer need to specify the statement ID,
and should therefore begin using CHANGE_CURRENT_STATEMENT_RUNTIME_PRIORITY
instead.
CHANGE_RUNTIME_PRIORITY will be removed in a future release. However, if you choose to
use this function, you can avoid specifying the query statement ID by setting the statement ID
value to NULL.
Administrator's Guide
See Also
l
CHANGE_CURRENT_STATEMENT_RUNTIME_PRIORITY
CHANGE_RUNTIME_PRIORITY
See Also
l
RESOURCE_POOL_DEFAULTS
Administrator's Guide
Administrator's Guide
Parameter
Guideline
MEMORYSIZE
EXECUTIONPARALLELISM
Administrator's Guide
PRIORITY
Valid values:
l
Administrator's Guide
RUNTIMEPRIORITY
Use this parameter to set the priority for running queries in a resource
pool. Any query with a duration that exceeds the
RUNTIMEPRIORITYTHRESHOLD value is assigned the run-time priority
you specify in this parameter.
l
Specifies the duration (in seconds) of queries that should always run
with HIGH run-time priority. Because all queries begin running with a
RUNTIMEPRIORITY of HIGH, queries that finish within the specified
threshold will run at a HIGH priority. All other queries are assigned the
run-time priority assigned to the resource pool.
Default value: 2
Disable setting: Set to 0.
QUEUETIMEOUT
Administrator's Guide
PLANNEDCONCURRENCY
Number of cores
Minimum value: 4.
Hewlett-Packard advises changing this value only after evaluating
performance over a period of time.
The Tuple Mover draws its resources from the TM pool. For the TM
pool, the PLANNEDCONCURRENCY parameter must be proportional to the
size of the RAM, the CPU, and the storage subsystem. Depending
on the storage type, if you increase PLANNEDCONCURRENCY for the
Tuple Mover threads, you might create storage I/O bottleneck.
Monitor the storage subsystem to determine if it becomes saturated
with long I/O queues. A saturated subsystem typically has more than
two I/O queues, and long latency in read and write.
If the storage subsystem is saturated, adjust the
PLANNEDCONCURRENCY parameter to keep the storage subsystem
resources below saturation level.
In addition, you might need to:
l
Administrator's Guide
Performance Considerations:
l
MAXCONCURRENCY
RUNTIMECAP
SINGLEINITIATOR
Administrator's Guide
CPUAFFINITYSET
CPUAFFINITYMODE
CASCADE TO
Usage Considerations
Use the various parameters to optimize your resource pools, but also monitor the state of the
following computational resources of your database cluster nodes:
l
CPU
RAM
Storage subsystems
Network bandwidth
If you have hardware or network limitations, configuring your resource pools may not completely
solve workload management issues.
See Also
l
Administrator's Guide
In all cases, you set this parameter as an interval value. The value cannot exceed one year. When
you set RUNTIMECAP for two or more of these levels, HP Vertica always uses the shortest value.
In this example, HP Vertica terminates any of user1's queries if they exceed 30 minutes. Even
though the user1's RUNTIMECAP is set for one hour, the pool on which the query is running only has a
30-minute RUNTIMECAP.
If a secondary pool for the ad_hoc_queries pool is specified using the CASCADE TO function, the
query will execute on that pool when the RUNTIMECAP on the ad_hoc_queries pool is surpassed.
See Also
l
RESOURCE_POOLS
Administrator's Guide
Solution
Create a separate resource pool for loads with a higher priority than the preconfigured setting on the
build-in GENERAL pool.
In this scenario, nightly loads get preference when borrowing memory from the GENERAL pool.
When loads are not running, all memory is automatically available for queries.
Note: If you are using the WOS, tune the PLANNEDCONCURRENCY parameter of the
WOSDATA pool to the number of concurrent loads. This ensures that AUTO spill to ROS is
configured in an optimal fashion.
Example
Create a resource pool with the PRIORITY of the pool set higher than the GENERAL pool.
For example, to create a pool designated for loads that has a higher priority then the GENERAL
pool, set load_pool with a priority of 10:
=> CREATE RESOURCE POOL load_pool PRIORITY 10;
Administrator's Guide
Solution
To ensure that a certain query or class of queries always gets resources, you could create a
dedicated pool for it as follows:
1. Using the PROFILE command, run the query that the CEO runs every week to determine how
much memory should be allocated:
=>
->
->
->
->
->
->
2. At the end of the query, the system returns a notice with resource usage:
NOTICE: Statement is being profiled.HINT: select * from v_monitor.execution_engine_
profiles where
transaction_id=45035996273751349 and statement_id=6;
NOTICE: Initiator memory estimate for query: [on pool general: 1723648 KB,
minimum: 355920 KB]
3. Create a resource pool with MEMORYSIZE reported by the above hint to ensure that the CEO
query has at least this memory reserved for it:
=> CREATE RESOURCE POOL ceo_pool MEMORYSIZE '1800M' PRIORITY 10;
CREATE RESOURCE POOL
=> \x
Expanded display is on.
=> SELECT * FROM resource_pools WHERE name = 'ceo_pool';
-[RECORD 1 ]-------+------------name
| ceo_pool
is_internal
| f
Administrator's Guide
memorysize
maxmemorysize
priority
queuetimeout
plannedconcurrency
maxconcurrency
singleinitiator
|
|
|
|
|
|
|
1800M
10
300
4
f
4. Assuming the CEO report user already exists, associate this user with the above resource pool
using ALTER USER statement.
=> ALTER USER ceo_user RESOURCE POOL ceo_pool;
5. Issue the following command to confirm that the ceo_user is associated with the ceo_pool:
=> SELECT * FROM users WHERE user_name ='ceo_user';
-[RECORD 1 ]-+-----------------user_id
| 45035996273713548
user_name
| ceo_user
is_super_user | f
resource_pool | ceo_pool
memory_cap_kb | unlimited
If the CEO query memory usage is too large, you can ask the Resource Manager to reduce it to fit
within a certain budget. See Target Memory Determination for Queries in Concurrent Environments.
Solution
User Profiles provides a solution to this scenario. To restrict the amount of memory Joe can use at
one time, set a MEMORYCAP for Joe to 100MB using the ALTER USER command. To limit the
amount of time that Joe's query can run, set a RUNTIMECAP to 2 hours using the same command.
If any query run by Joe takes up more than its cap, HP Vertica rejects the query.
If you have a whole class of users whose queries you need to limit, you can also create a resource
pool for them and set RUNTIMECAP for the resource pool. When you move these users to the
Administrator's Guide
resource pool, HP Vertica limits all queries for these users to the RUNTIMECAP you specified for
the resource pool.
Example
=> ALTER USER analyst_user MEMORYCAP '100M' RUNTIMECAP '2 hours';
If Joe attempts to run a query that exceeds 100MB, the system returns an error that the request
exceeds the memory session limit, such as the following example:
\i vmart_query_04.sqlvsql:vmart_query_04.sql:12: ERROR: Insufficient resources to
initiate plan
on pool general [Request exceeds memory session limit: 137669KB > 102400KB]
Only the system database administrator (dbadmin) can increase only the MEMORYCAP setting.
Users cannot increase their own MEMORYCAP settings and will see an error like the following if
they attempt to edit their MEMORYCAP or RUNTIMECAP settings:
ALTER USER analyst_user MEMORYCAP '135M';
ROLLBACK: permission denied
Solution
The simplest solution is to create a standalone resource pool for the ad hoc applications so that the
total MEMORYSIZE is fixed. Recall that in a standalone pool, MAXMEMORYSIZE is set equal to
MEMORYSIZE so no memory can be borrowed from the GENERAL pool. Associate this user pool
with the database user(s) from which the application uses to connect to the database. In addition,
set RUNTIMECAP to limit the maximum duration of an ad hoc query.
Other solutions include limiting the memory usage of individual users such as in the Scenario:
Preventing Run-Away Queries.
Tip: Besides adding limits such as the above, it is also a great idea to train the user community
on writing good SQL.
Administrator's Guide
Example
To create a standalone resource pool for the ad hoc users, set the MEMORYSIZE equal to the
MAXMEMORYSIZE:
=> CREATE RESOURCE POOL adhoc_pool MEMORYSIZE '200M' MAXMEMORYSIZE '200M'
QUEUETIMEOUT 300 PLANNEDCONCURRENCY 4;
=> SELECT pool_name, memory_size_kb, queueing_threshold_kb
FROM V_MONITOR.RESOURCE_POOL_STATUS w
WHERE is_standalone = 'true' AND is_internal = 'false';
pool_name | memory_size_kb | queueing_threshold_kb
------------+----------------+----------------------adhoc_pool |
204800 |
153600
(1 row)
PRIORITY 0
After the pool has been created, associate the ad hoc users with the adhoc_pool:
=> ALTER USER app1_user RESOURCE POOL adhoc_pool;=> ALTER RESOURCE POOL adhoc_pool
MEMORYSIZE '10M' MAXMEMORYSIZE '10M';
\i vmart_query_04.sql
vsql:vmart_query_04.sql:12: ERROR: Insufficient resources to initiate plan
on pool adhoc_pool [Request Too Large:Memory(KB)
Exceeded: Requested = 84528, Free = 10240 (Limit = 10240, Used = 0)]
The query will not borrow memory from the GENERAL pool and gets rejected with a 'Request Too
Large' message.
Solution
The simplest solution is to create a separate resource pool for the users of that application and set
its MAXCONCURRENCY to the desired concurrency level. Any queries beyond
MAXCONCURRENCY are queued.
Tip: HP recommends leaving PLANNEDCONCURRENCY to the default level so the queries
get their maximum amount of resources. The system as a whole thus runs with the highest
efficiency.
Administrator's Guide
Example
In this example, there are four billing users associated with the billing pool. The objective is to set a
hard limit on the resource pool so a maximum of three concurrent queries can be executed at one
time. All other queries will queue and complete as resources are freed.
=> CREATE RESOURCE POOL billing_pool MAXCONCURRENCY 3 QUEUETIMEOUT 2;
=> CREATE USER bill1_user RESOURCE POOL billing_pool;
=> CREATE USER bill2_user RESOURCE POOL billing_pool;
=> CREATE USER bill3_user RESOURCE POOL billing_pool;
=> CREATE USER bill4_user RESOURCE POOL billing_pool;
=> \x
Expanded display is on.
=> select maxconcurrency,queuetimeout from resource_pools where name = 'billing_pool';
maxconcurrency | queuetimeout
----------------+-------------3 |
2
(1 row)
> SELECT reason, resource_type, rejection_count FROM RESOURCE_REJECTIONS
WHERE pool_name = 'billing_pool' AND node_name ilike '%node0001';
reason
| resource_type | rejection_count
---------------------------------------+---------------+----------------Timedout waiting for resource request | Queries
|
16
(1 row)
If queries are running and do not complete in the allotted time (default timeout setting is 5 minutes),
the next query requested gets an error similar to the following:
ERROR: Insufficient resources to initiate plan on pool billing_pool [Timedout waiting
for resource request: Request exceeds limits:
Queries Exceeded: Requested = 1, Free = 0 (Limit = 3, Used = 3)]
The table below shows that there are three active queries on the billing pool.
=> SELECT pool_name, thread_count, open_file_handle_count, memory_inuse_kb FROM RESOURCE_
ACQUISITIONS
WHERE pool_name = 'billing_pool';
pool_name
--------------+--------------+------------------------+----------------billing_pool |
4 |
5 |
132870
billing_pool |
4 |
5 |
132870
billing_pool |
4 |
5 |
132870
(3 rows)
Administrator's Guide
Solution
The principles learned from the previous scenarios can be applied to solve this problem. The basic
idea is to segregate the queries into two groups associated with different resource pools. The
prerequisite is that there are two distinct database users issuing the different types of queries. If
this is not the case, do consider this a best practice for application design.
METHOD 1: Create a dedicated pool for the web page refresh queries where you:
1. Size the pool based on the average resource needs of the queries and expected number of
concurrent queries issued from the portal.
2. Associate this pool with the database user that runs the web site queries. (See Scenario: The
CEO Query for detailed procedure on creating a dedicated pool.)
3. This ensures that the web site queries always run and never queue behind the large batch jobs.
Leave the batch jobs to run off the GENERAL pool.
For example, the following pool is based on the average resources needed for the queries
running from the web and the expected number of concurrent queries. It also has a higher
PRIORITY to the web queries over any running batch jobs and assumes the queries are being
tuned to take 250M each:
CREATE RESOURCE POOL web_pool MEMORYSIZE '250M' MAXMEMORYSIZE NONE PRIORITY 10
MAXCONCURRENCY 5 PLANNEDCONCURRENCY 1
METHOD 2: Create a standalone pool to limit the batch reports down to a fixed memory size so
memory is always left available for other purposes. (See Scenario: Restricting Resource Usage of
Ad Hoc Query Application.)
For example:
CREATE RESOURCE POOL batch_pool MEMORYSIZE '4G' MAXMEMORYSIZE '4G' MAXCONCURRENCY 10:
Administrator's Guide
The same principle can be applied if you have three or more distinct classes of workloads.
Solution
The solution is similar to the mixed workload case. In this scenario, you do not limit resource usage;
you set different priorities. To do so, create two different pools, each with MEMORYSIZE=0% and
a different PRIORITY parameter. Both pools borrow from the GENERAL pool, however when
competing for resources, the priority determine the order in which each pool's request is granted.
For example:
=> CREATE RESOURCE POOL dept1_pool PRIORITY 5;
=> CREATE RESOURCE POOL dept2_pool PRIORITY 8;
If you find this solution to be insufficient, or if one department's queries continuously starves
another departments users, you could add a reservation for each pool by setting MEMORYSIZE so
some memory is guaranteed to be available for each department.
For example, since both resources are using the GENERAL pool for memory, you could allocate
some memory to each resource pool by using the ALTER RESOURCE POOL command to change
the MEMORYSIZE for each pool:
=> ALTER RESOURCE POOL dept1_pool MEMORYSIZE '100M';
=> ALTER RESOURCE POOL dept2_pool MEMORYSIZE '150M';
Solution
The solution to this scenario will depend on your query mix; however, below are the general steps to
take:
Administrator's Guide
1. Determine the number of continuous load streams required. This may be related to the desired
load rate if a single stream does not provide adequate throughput, or may be more directly
related to the number of sources of data to load. Also determine if automatic storage is best, or
if DIRECT is required. Create a dedicated resource pool for the loads, and associate it with the
database user that will perform them. See CREATE RESOURCE POOL for details.
In general, the concurrency settings for the load pool should be less than the number of cores
per node. Unless the source processes are slow, it is more efficient to dedicate more memory
per load, and have additional loads queue. Adjust the load pool's QUEUETIMEOUT setting if
queuing is expected.
2. If using automatic targeting of COPY and INSERT, set the PLANNEDCONCURRENCY
parameter of the WOSDATA pool to the number of concurrent loads expected. Also, set
MEMORYSIZE of the WOS to the expected size of the loaded data to ensure that small loads
don't spill to ROS immediately. See Built-In Pools for details.
3. Run the load workload for a while and observe whether the load performance is as expected. If
the Tuple Mover is not tuned adequately to cover the load behavior, see Tuning the Tuple
Mover in Administrator's Guide.
4. If there is more than one kind of query in the system (say some queries that must be answered
quickly for interactive users, and others that are part of a batch reporting process), follow the
advice in Scenario: Handling Mixed Workloads.
5. Let the queries run and observe the performance. If some classes of queries are not getting the
desired performance, then it may be necessary to tune the GENERAL pool as outlined in
Scenario: Restricting Resource Usage of Ad Hoc Query Application, or to create further
dedicated resource pools for those queries. See Scenario: The CEO Query and Scenario:
Handling Mixed Workloads.
See the sections on Managing Workloads and CREATE RESOURCE POOL for additional details
and tips for obtaining predictable results in mixed workload environments.
Administrator's Guide
now you want to manage resources at run time and ensure that short queries always run with a high
priority and are never queued as a result of limited run-time resources.
Solution
Set the RUNTIMEPRIORITY for the resource pool to MEDIUM or LOW. Set the
RUNTIMEPRIORITYTHRESHOLD for the resource pool to the duration of queries you want to ensure
always run at a high priority. For instance, if you set this value to 5, all queries that complete within
5 seconds will run at high priority. Any other query that exceeds 5 seconds will drop down to the
RUNTIMEPRIORITY assigned to the resource pool (MEDIUM or LOW).
Example
To ensure that all queries with a duration of less than 5 seconds always run at a high priority, modify
adhoc_pool as follows:
l
Solution
Set the RUNTIMEPRIORITY for the resource pool to LOW and set the
RUNTIMEPRIORITYTHRESHOLD to a number that cuts off only the longest jobs.
Example
To ensure that all queries with a duration of more than 3600 seconds (1 hour) are assigned a low
runtime priority, modify the resource pool as follows:
l
Administrator's Guide
Administrator's Guide
Solution
Set the MAXMEMORYSIZE parameter of the GENERAL pool to the desired memory size. See
Resource Pool Architecture for a discussion on resource limits.
Solution
Set the PLANNEDCONCURRENCY and MAXCONCURRENCY setting of the recovery pool to 1
so that recovery can take as much memory as possible from the GENERAL pool and run only one
thread at once.
Note: This setting could slow down other queries in your system.
Administrator's Guide
Solution
Set the MEMORYSIZE parameter of the refresh pool to a fixed value. The Resource Manager then
tunes the refresh query to only use this amount of memory.
Tip: Remember to reset the refresh pool MEMORYSIZE back to 0% after the refresh operation
completes so memory can be used for other operations.
Solution
Increase the MAXCONCURRENCY parameter of the TM pool to 3 or higher. This setting ensures
that the Tuple Mover can run more than one mergeout thread, so if a large mergeout is in progress,
smaller ROS containers can also be merged, thus preventing a buildup of ROS containers.
Examining the query plan usually reveals one or more more of the following:
Administrator's Guide
Note: Although you cannot see that a partitioned hash join occurred in the plan, you can see
that when the optimizer chose a hash join.
See Creating Custom Designs to understand projection design techniques. The Database Designer
automatically applies these techniques to suggest optimal designs for queries.
Real-Time Profiling
HP Vertica provides profiling mechanisms that let you determine how well the database is
performing. For example, HP Vertica can collect profiling data for a single statement, a single
session, or for all sessions on all nodes.
Real-time profiling is always "on", without profiling being explicitly enabled.
For details, see:
l
See Also
l
EXECUTION_ENGINE_PROFILES
Administrator's Guide
Description
RESOURCE_
REJECTIONS
Manager.
DISK_
Monitors requests for resources that are rejected due to disk space
RESOURCE_
REJECTIONS
When requests for resources of a certain type are being rejected, do one of the following:
l
Increase the resources available on the node by adding more memory, more disk space, and so
on. See Managing Disk Space.
Reduce the demand for the resource by reducing the number of users on the system (see
Managing Sessions), rescheduling operations, and so on.
The message Usage of a single requests exceeds high limit means that the system
does not have enough of the resource available for the single request. A common example
occurs when the file handle limit is set too low and you are loading a table with a large number of
columns.
The message Timed out or Canceled waiting for resource reservation usually
means that there is too much contention for the resource because the hardware platform cannot
support the number of concurrent users using it.
See Also
l
Administrator's Guide
Managing Sessions
HP Vertica provides powerful methods for database administrators to view and control sessions.
The methods vary according to the type of session:
l
External (user) sessions are initiated by vsql or programmatic (ODBC or JDBC) connections
and have associated client state.
Internal (system) sessions are initiated by the HP Vertica database process and have no client
state.
You can view a list of currently active sessions (including internal sessions) and can interrupt or
close external sessions when necessary, particularly when shutting down the database.
By default HP Vertica allows 50 client sessions and an additional five administrator sessions per
node.You can configure the maximum number of client sessions that can run on a single database
cluster node using the MaxClientSessions parameter.HP Vertica always allows five
administrative sessions per node in addition to the value set using the MaxClientSession
parameter.
Important: If you use the Administration Tools "Connect to Database" option, HP Vertica will
attempt connections to other nodes if a local connection does not succeed. These cases can
result in more successful "Connect to Database" commands than you would expect given the
MaxClientSessions value.
To increase the number of MaxClientSessions to 100, issue the following command at a vsql
prompt:
=> ALTER DATABASE mydb SET MaxClientSessions = 100;
Viewing Sessions
HP Vertica provides the SESSIONS table to view the session status of your database. SESSIONS
contains information about external sessions and returns one row per session. This table is
described in the SQL Reference Manual.
Administrator's Guide
Note: Superuser has unrestricted access to all database metadata. Users have significantly
reduced access to metadata based on their privileges. See Metadata Privileges.
Closing a session interrupts the session and disposes of all state related to the session,
including client socket connections for external sessions.
These actions are provided in the form of SQL functions, described in the SQL Reference Manual:
l
INTERRUPT_STATEMENT
CLOSE_SESSION
CLOSE_ALL_SESSIONS
SHUTDOWN
SELECT statements that call these functions return when the interrupt or close message has been
delivered to all nodes, not after the interrupt or close has completed. This means there might be a
delay after the statement returns and the interrupt or close taking effect throughout the cluster. To
determine if the session or transaction has ended, you can monitor the SESSIONS system table.
Controlling Sessions
The database administrator must be able to disallow new incoming connections in order to shut
down the database. On a busy system, database shutdown is prevented if new sessions connect
after the CLOSE_SESSION or CLOSE_ALL_SESSIONS() command is invokedand before the
database actually shuts down.
One option is for the administrator to issue the SHUTDOWN('true') command, which forces the
database to shut down and disallow new connections. See SHUTDOWN in the SQL Reference
Manual.
Another option is to modify the MaxClientSessions parameter from its original value to 0, in order
to prevent new non-dbadmin users from connecting to the database.
Administrator's Guide
1. Determine the original value for the MaxClientSessions parameter by querying the V_
MONITOR.CONFIGURATIONS_PARAMETERS system table:
=> SELECT CURRENT_VALUE FROM CONFIGURATION_PARAMETERS WHERE parameter_
name='MaxClientSessions';
CURRENT_VALUE
--------------50
(1 row)
When the session no longer appears in the SESSIONS table, disconnect and run the Stop
Database command.
5. Restart the database.
6. Restore the MaxClientSessions parameter to its original value:
=> ALTERDATABASE mydb SET MaxClientSessions = 50;
Administrator's Guide
SQL Reference Manual
Administrator's Guide
See Also
l
Configuration Parameters
Stopping a Database
CONFIGURATION_PARAMETERS
CLOSE_ALL_SESSIONS
CLOSE_SESSION
INTERRUPT_STATEMENT
SESSIONS
SHUTDOWN
Description
LOAD_STREAMS Monitors load metrics for each load stream on each node.
For details about HP Vertica system tables, see HP Vertica System Tables.
When a COPY statement using the DIRECT option is in progress, the ACCEPTED_ROW_COUNT value can
increase to the maximum number of rows in the input file as the rows are being parsed.
If COPY reads input data from multiple named pipes, the PARSE_COMPLETE_PERCENT value will
remain at zero (0) until all named pipes return an EOF. While COPY awaits an EOF from multiple
pipes, it can appear to be hung. However, before canceling the COPY statement, check your
system CPU and disk accesses to see if any activity is in progress.
In a typical load, the PARSE_COMPLETE_PERCENT value can either increase slowly to 100%, or jump
to 100% quickly if you are loading from named pipes or STDIN, while SORT_COMPLETE_PERCENT is
at 0. Once PARSE_COMPLETE_PERCENT reaches 100%, SORT_COMPLETE_PERCENT increases to
100%. Depending on the data sizes, a significant lag can occur between the time PARSE_
COMPLETE_PERCENT reaches 100% and the time SORT_COMPLETE_PERCENT begins to increase.
Administrator's Guide
Managing Storage Locations
Local storage locationsStore data in a unique location for each node. This location is in a
directory in a file system that the node can access, and is often in the nodes own file system.
You can create a local storage location for a single node or for all nodes in the cluster. Clusterwide storage locations are the most common type of storage. HP Vertica defaults to using a
local cluster-wide storage location for storing all data. If you want it to store data differently, you
must create additional storage locations.
Shared storage locations Based on a single file system to which all nodes in the cluster have
access. This shared file system is often hosted outside of the cluster, such as a Hadoop
Distributed File System (HDFS), which ishosted on a Hadoop cluster. When you create a
shared storage location, each node in the HP Vertica cluster creates its own subdirectory in the
location. These separate directories prevent the nodes from overwriting each others data.
Currently, HP Verticasupports only shared storage locations on HDFS. See HP Vertica Storage
Location for HDFS in the Hadoop Integration Guide for more information.
Administrator's Guide
Managing Storage Locations
Place temp space directly on the local Linux file system, not in HDFS.
For the best performance, place the HP Vertica catalog directly on the local Linux file system.
Create the database first on a local Linux file system. Then, you can extend the database to
HDFS storage locations and set storage policies that exclusively place data blocks on the
HDFS storage location.
For better performance, do not run the HDFSbalancer on HP Vertica nodes. The
HDFSbalancer can move data blocks farther away, causing HP Vertica to read non-local data
during query execution. Queries run faster if they do not require network I/O.
Generally, HDFS requires approximately 2 GB of memory for each node in the cluster. To support
this requirement in your HP Vertica configuration:
1. Create a 2-GB resource pool.
2. Do not assign any HP Vertica execution resources to this pool. This approach reserves the
space for use by HDFS.
For more about how to configure resource pools, see Managing Workloads.
Administrator's Guide
Managing Storage Locations
Note: You cannot add or remove a catalog storage location. HP Vertica creates and manages
this storage location internally, and the area exists in the same location on each cluster node.
storage_containers
storage_locations
partitions
Use a query such as the following for relevant columns of the storage_containers system table:
VMART=> select node_name,projection_name, location_label from v_monitor.storage_
containers;
node_name
|
projection_name
| location_label
------------------+----------------------+----------------v_vmart_node0001 | states_p_node0001
|
v_vmart_node0001 | states_p_node0001
|
v_vmart_node0001 | t1_b1
|
v_vmart_node0001 | newstates_b0
| FAST3
v_vmart_node0001 | newstates_b0
| FAST3
v_vmart_node0001 | newstates_b1
| FAST3
v_vmart_node0001 | newstates_b1
| FAST3
v_vmart_node0001 | newstates_b1
| FAST3
.
.
.
Use a query such as the following for columns of the v_catalog.storage_locations system_
table:
VMart=> select node_name, location_path, location_usage, location_label from storage_
locations;
node_name
|
location_path
| location_usage |
location_label
------------------+-------------------------------------------+----------------+--------------v_vmart_node0001 | /home/dbadmin/VMart/v_vmart_node0001_data | DATA,TEMP
|
v_vmart_node0001 | home/dbadmin/SSD/schemas
| DATA
|
v_vmart_node0001 | /home/dbadmin/SSD/tables
| DATA
| SSD
v_vmart_node0001 | /home/dbadmin/SSD/schemas
| DATA
| Schema
v_vmart_node0002 | /home/dbadmin/VMart/v_vmart_node0002_data | DATA,TEMP
|
v_vmart_node0002 | /home/dbadmin/SSD/tables
| DATA
|
v_vmart_node0002 | /home/dbadmin/SSD/schemas
| DATA
|
v_vmart_node0003 | /home/dbadmin/VMart/v_vmart_node0003_data | DATA,TEMP
|
Administrator's Guide
Managing Storage Locations
v_vmart_node0003 | /home/dbadmin/SSD/tables
v_vmart_node0003 | /home/dbadmin/SSD/schemas
(10 rows)
| DATA
| DATA
|
|
Use a query such as the following for columns of the v_monitor.partitions system table:
VMART=> select partition_key, projection_name, location_label from v_monitor.partitions;
partition_key |
projection_name
| location_label
---------------+----------------------+--------------NH
| states_b0
| FAST3
MA
| states_b0
| FAST3
VT
| states_b1
| FAST3
ME
| states_b1
| FAST3
CT
| states_b1
| FAST3
.
.
.
Administrator's Guide
Managing Storage Locations
single node to all cluster nodes. However, do not use a shared directory on one node for other
cluster nodes to access.
You can add and configure storage locations (other than the required defaults) to provide additional
storage for these purposes:
l
TEMP Temporary files that are generated and dumped to disk, such as those generated
by sort, group by, join, and so on.
USER Gives access to non-dbadmin users so they can use the storage location after
being granted read or write privileges. You cannot assign this location type for use in a
storage policy.
Tip: Storing temp and data files in different storage locations is advantageous because
the two types of data have different disk I/O access patterns. Temp files are distributed
across locations based on available storage space. However, standard data files can be
stored on different storage locations, based on predicted or measured access patterns.
Administrator's Guide
Managing Storage Locations
On each node in the cluster, create a directory where the node stores its data. For example:
$mkdir /home/dbadmin/vertica/SSD
HP recommends that you create the same directory path on each node. Use this path when
creating a storage location.
Use the CREATE LOCATION statement to add a storage location.
1. Specify the following information:
n
Important: Specify the node or use the ALL NODES keyword. Otherwise, the
statement creates the storage locations on all nodes in the cluster in a single
transaction.
n
For user access (non-dbadmin users), you must create the storage location with the USER
usage type. You cannot change an existing storage location to have USER access. After you
Administrator's Guide
Managing Storage Locations
create a USER storage location, you can grant one or more users access to the area. User
areas can store only data files, not temp files. You cannot assign a USER storage location to a
storage policy.
The following example shows how to add a location available on all nodes to store only data:
CREATELOCATION '/home/dbadmin/vertica/SSD/' ALLNODESUSAGE 'DATA';
The following example shows how to add a location that is available on node v_vmart_
node0001 to store data and temporary files:
CREATELOCATION '/home/dbadmin/vertica/SSD/' NODE'v_vmart_node0001';
2. Suppose you are using a storage location for data files and want to create ranked storage
locations. In this ranking, columns are stored on different disks based on their measured
performance. To do so, you should first:
a. Measure the performance of the storage location.
b. Set the performance of the storage location.
Note: After you create a storage location, you can alter the type of information it stores, with
some restrictions. See Altering Storage Location Use.
Administrator's Guide
Managing Storage Locations
This example shows you how to add a new DATA storage location with a label, SSD. The label
identifies the location when you create storage policies. Specifying the ALLNODES keyword adds
the storage location to all nodes in a single transaction:
=> CREATE LOCATION '/home/dbadmin/SSD/schemas' ALL NODES USAGE 'DATA'
LABEL 'SSD';
Note: A data location can already exist and be in use before you identify it as a storage location
with the function. For instance, as a superuser, you can set up an area in which to store and
test external tables. When you are ready for user BOB to access the location, you add a storage
Administrator's Guide
Managing Storage Locations
location at the path you used for testing and grant BOB the required privileges.
The following example shows how to grant a user BillyBob READ/WRITE permissions to the
/BillyBobStore location:
=> GRANT ALL ON LOCATION '/home/dbadmin/UserStorage/BillyBobStore' TO BillyBob;
GRANT PRIVILEGE
For more information about configuring user privileges, see Managing Users and Privileges in the
Administrator's Guide and the GRANT (Storage Location) and REVOKE (Storage Location)
functions in the SQL Reference Manual.
Storage locations created with the USER option can store only DATA files.
Administrator's Guide
Managing Storage Locations
storage locations.
Altering an existing storage location has the following effects:
To store only:
Has this
effect:
Temp files
Data
content is
eventually
merged out
through
ATM, per its
policies.You
can also
merge out
data from
the storage
location
manually
using DO_
TM_TASK.
The location
stores only
temp files
from that
point
forward.
Administrator's Guide
Managing Storage Locations
To store only:
Has this
effect:
Data files
HP Vertica
continues to
run all
statements
that use
temp files
(such as
queries and
loads).
Subsequent
statements
no longer
use the
changed
storage
location for
temp files,
and the
location
stores only
data files
from that
point
forward.
Remove a label.
Perform cluster-wide operations, if you also specify an empty string for the function's node
parameter ('').
Administrator's Guide
Managing Storage Locations
Important: You can label an existing storage location that already contains data.
However, including the labeled location in one or more storage policies can the ATMto
relocate existing data. This situation occurs if the ATM determines data stored on a
labeled location does not comply with a storage policy.
The location from which you are removing the label is the last available storage for its associated
objects
The next example removes the SSD label for the storage location on all nodes by specifying empty
strings ('') for both node and location_label parameters:
VMART=> select alter_location_label('/home/dbadmin/SSD/tables','', '');
alter_location_label
------------------------------------------
Administrator's Guide
Managing Storage Locations
Note:
Alter label:
To:
Has this
effect:
No name
New label
Lets you
use the
labeled
storage
within a
storage
policy.
See note
above
regarding
data
storage
being
moved to
other
locations
if you add
a label to
a storage
location
with
existing
data.
Administrator's Guide
Managing Storage Locations
Alter label:
To:
Has this
effect:
Existing name
New name
You can
use the
new label
in a
storage
policy. If
the
existing
name is
used in a
storage
policy,
you
cannot
change
the label.
Existing name
No name
You
cannot
use an
unlabeled
storage
in a
storage
policy. If
the
existing
name is
used in a
storage
policy,
you
cannot
remove
the label.
Administrator's Guide
Managing Storage Locations
Administrator's Guide
Managing Storage Locations
YES
Lower
STANDARD
NO
Medium
N/A
YES
Higher/highest
FAST
NO
Highest
N/A
Income (table)
Lower
Storage policies exist for the database and table objects, with default storage on the locations
STANDARD and FAST, respectively.
In this case, when TM operations occur, such as moveout and mergeout, table data has the highest
priority. The TM moves data from WOS to ROS to the FAST labeled location.
Any schema data changes are prioritized after table data. Because the Region schema has no
storage policy, HP Vertica searches up the storage levels for a policy. In this case, that is the
Sales database itself. If a database storage policy is in effect, Region schema data is moved from
WOS to ROS to the STANDARD storage location, using its parent object's default storage location.
If the Sales database object had no storage policy, the TM operations would use existing storage
locations and mechanisms.
Administrator's Guide
Managing Storage Locations
(1 row)
You can query existing storage policies, listed in the location_label column of the v_
monitor.storage_containers system table:
VMART=> select node_name, projection_name, storage_type, location_label from v_
monitor.storage_containers;
node_name
|
projection_name
| storage_type | location_label
------------------+----------------------+--------------+--------------v_vmart_node0001 | states_p_node0001
| ROS
|
v_vmart_node0001 | states_p_node0001
| ROS
|
v_vmart_node0001 | t1_b1
| ROS
|
v_vmart_node0001 | newstates_b0
| ROS
| LEVEL3
v_vmart_node0001 | newstates_b0
| ROS
| LEVEL3
v_vmart_node0001 | newstates_b1
| ROS
| LEVEL3
v_vmart_node0001 | newstates_b1
| ROS
| LEVEL3
v_vmart_node0001 | newstates_b1
| ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_p_v1_node0001 | ROS
| LEVEL3
v_vmart_node0001 | states_b0
| ROS
| SSD
v_vmart_node0001 | states_b0
| ROS
| SSD
v_vmart_node0001 | states_b1
| ROS
| SSD
v_vmart_node0001 | states_b1
| ROS
| SSD
v_vmart_node0001 | states_b1
| ROS
| SSD
.
.
.
Storage effect:
Database The highest storage level and the lowest storage priority. This is the default policy
when no lower-level or higher priority policies exist. At storage time, HP Vertica uses
the database policy for all objects without storage policies.
Schema
The mid-level storage, also with a medium priority, compared to lower storage level
objects. If a table's schema has no policy, the TM searches the next higher level, the
database, using that policy, if it exists. If it does not, the TM uses existing storage
mechanisms.
Administrator's Guide
Managing Storage Locations
Create
policy
for...
Storage effect:
Table
A lower storage level than a schema, with the highest storage priority, if no policy
exists for the table's partition key ranges. If a table has no storage policy, HP Vertica
checks the next higher storage level (the schema) for a policy and uses that. If the
schema has no policy, it checks the next higher level, the database, and uses that. If
no database policy exists, the TM uses existing storage mechanisms.
Partition
The lowest level policy that can be in effect. During storage processing, partition key
min_key
ranges with a policy have the highest priority. If no policy exists, the parent table is
and max_ checked, and so on as described for the other database objects.
key
ranges
2. Move all existing data for the specified objects to the target storage location.
HP Vertica moves the existing data the next time the Tuple Mover runs. Alternatively, you can
enforce the data move to occur in the current transaction using the function's enforce_storage_
move parameter. If the storage location is remote(for example, in HDFS), Hewlett-Packard
recommends setting this parameter to true.
Before actually moving the object to the target storage location, HP Vertica calculates the required
storage and checks available space at the target. If there is insufficient free space, the function
generates an error and stops execution. It does not attempt to find sufficient storage at another
location.
Important: You should check available space on the new target location before starting a data
move. However, be aware that checking does not guarantee that this space remains available
Administrator's Guide
Managing Storage Locations
during move execution. Checking target space does prevent attempts to move any data, if
insufficient space is available.
Administrator's Guide
Managing Storage Locations
Object type:
Effect:
Schema or table
If data storage exists, moves data from source to target destination. Source
data can reside on a labeled or unlabeled storage location, but will be moved
to specified labeled location.
Cluster nodes unavailable when existing data is copied are updated by the
TM when they rejoin the cluster. Alternately, you can enforce a data moveas
part of the function transaction, by specifying the last parameter as true.
If a storage policy was in effect, the default storage location changes from
the source to target location. This change affects all future TM operations,
such as moveout and mergeout activities.
Table with
Sets a policy or moves existing data only for the key_min and key_max
specified
ranges. Separate partition key ranges can have different storage policies
partition min-
Existing data is moved to the parent storage policy's location, or the default storage location if there
is no parent policy. Typically, the move occurs the next time the Tuple Mover runs. Alternatively,
you can force the data move to occur in the current transaction by setting the optional enforce_
storage_move parameter to true.
You can also use the ENFORCE_OBJECT_STORAGE_POLICY meta-function to trigger the
move for all storage locations at once. Using this function equates to setting enforce_storage_
move.
Administrator's Guide
Managing Storage Locations
For this example, clearing the storage policy for an objected named lineorder, removes the policy
for the table, while retaining storage policies for its partitions, which have their own policies.
The function determines which lineorder object policy to clear because no partition range values
are specified in the function call:
release=> select clear_object_storage_policy('lineorder');
clear_object_storage_policy
------------------------------------Default storage policy cleared.
(1 row)
release=> select * from v_monitor.storage_policies;
schema_name | object_name | policy_details | location_label
-------------+-------------+------------------+---------------| public | Schema | F4
public | lineorder | Partition [0, 0] | F1
public | lineorder | Partition [1, 1] | F2
public | lineorder | Partition [2, 2] | F4
public | lineorder | Partition [3, 3] | M1
public | lineorder | Partition [4, 4] | M3
(6 rows)
Further, using a partition key range with the lineorder object name clears the storage policy for
only the specified partition range(s). The storage policy for the parent table objects, and other
partition ranges persist:
release=> select clear_object_storage_policy ('lineorder','0','3');
clear_object_storage_policy
-------------------------------------
Administrator's Guide
Managing Storage Locations
Latency is for random reads only in seeks (units in seeks per second).
Thus, the I/O time of a faster storage location is less than that of a slower storage location.
Note: Measuring storage location performance requires extensive disk I/O, which is a
resource-intensive operation. Consider starting this operation when fewer other operations are
running.
HP Vertica gives you two ways to measure storage location performance, depending on whether
the database is running. You can either:
l
Administrator's Guide
Managing Storage Locations
Both methods return the throughput and latency for the storage location. Record or capture the
throughput and latency information so you can use it to set the location performance (see Setting
Storage Performance).
You need RAM*2 free space available in a storage location to measure its performance. For
example, if you have 16 GB RAM, you need 32 GB of available disk space. If you do not have
enough disk space, the function returns an error.
Use the system table DISK_STORAGE to obtain information about disk storage on each database
node.
The following example shows how to measure the performance of a storage location on v_
vmartdb_node0004:
=> SELECT MEASURE_LOCATION_PERFORMANCE('/secondVerticaStorageLocation/','v_vmartdb_
node0004');
WARNING: measure_location_performance can take a long time. Please check logs for
progress
measure_location_performance
-------------------------------------------------Throughput : 122 MB/sec. Latency : 140 seeks/sec
For example:
opt/vertica/bin/vertica -m /secondVerticaStorageLocation/node0004_data
Administrator's Guide
Managing Storage Locations
Columns in the sort order are given the highest priority (numbers > 1000).
The last column in the sort order is given the rank number 1001.
The next-to-last column in the sort order is given the rank number 1002, and so on until the first
column in the sort order is given 1000 + # of sort columns.
The remaining columns are given numbers from 10001, starting with 1000 and decrementing by
one per column.
HP Vertica then stores columns on disk from the highest ranking to the lowest ranking. It places
highest-ranking columns on the fastest disks and the lowest-ranking columns on the slowest disks.
Administrator's Guide
Managing Storage Locations
Set the locations with the highest performance as the default locations for critical data.
Use slower locations as default locations for older, or less-important data. Such slower locations
may not require policies at all, if you do not want to specify default locations.
HP Vertica determines data storage as follows, depending on whether a storage policy exists:
Storage
Number
Policy
Label ofLocations
HP Vertica Action
No
No
Multiple
Yes
Single
Yes
Yes
Multiple
When you drop a storage location, the operation cascades to associated objects including any
granted privileges to the storage.
Caution: Dropping a storage location is a permanent operation and cannot be undone.
Subsequent queries on storage used for external table access fail with a COPY COMMAND
FAILED message.
Administrator's Guide
Managing Storage Locations
Because dropping a storage location cannot be undone, HP recommends that you first retire a
storage location (see Retiring Storage Locations). Retiring a storage location before dropping it lets
you verify that there will be no adverse effects on any data access. Additionally, you can restore a
retired storage location (see Restoring Retired Storage Locations).
Retire the location, and force the changes to take effect immediately.
To retire a storage location on all nodes, use an empty string ('') for the second parameter.
Administrator's Guide
Managing Storage Locations
Retiring a location prevents HP Vertica from storing data or temp files to it, but does not remove the
actual location. Any data previously stored on the retired location is eventually merged out by the
Automatic Tuple Mover (ATM) per its policies.
If you plan to drop the storage location after retiring it, you can expedite this by setting the optional
enforce_storage_move parameter to true. This setting moves the data out of the storage location
instead of waiting for the Tuple Mover, allowing you to drop the location immediately.
You can also use the ENFORCE_OBJECT_STORAGE_POLICY meta-function to trigger the
move for all storage locations at once, and then drop the locations. This approach equates to setting
enforce_storage_move.
The following example shows how to retire a storage location on all nodes and prepares it for
immediate drop:
=> SELECT RETIRE_LOCATION('/secondStorageLocation/' , '', true);
Note: If the location used in a storage policy is the last available storage for its associated
objects, you cannot retire it unless you set enforce_storage_move to true.
Data and temp files can be stored in one, or multiple separate, storage locations.
For further information on dropping a location after retiring it, see Dropping Storage Locations.
To restore a storage location on all nodes, use an empty string ('') for the second parameter. The
following example demonstrates creating, retiring, and restoring a location on all nodes:
=> create location '/tmp/foo' all nodes usage 'TEMP';
CREATE LOCATION
=> select retire_location('/tmp/foo', '');
retire_location
Administrator's Guide
Managing Storage Locations
-----------------------/tmp/foo retired.
(1 row)
=> select * from storage_locations where location_path ilike '/tmp/foo';
location_id
| node_name | location_path | location_usage | is_retired | location_label
| rank | throughput | latency
-------------------+-----------+---------------+----------------+------------+---------------+------+------------+--------45035996273736724 | node01
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
45035996273736726 | node02
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
45035996273736728 | node03
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
45035996273736730 | node04
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
(4 rows)
=> select restore_location('/tmp/foo', '');
restore_location
------------------------/tmp/foo restored.
(1 row)
=> select * from storage_locations where location_path ilike '/tmp/foo';
location_id
| node_name | location_path | location_usage | is_retired | location_label
| rank | throughput | latency
-------------------+-----------+---------------+----------------+------------+---------------+------+------------+--------45035996273736724 | node01
| /tmp/foo
| TEMP
| f
|
|
0 |
0 |
0
45035996273736726 | node02
| /tmp/foo
| TEMP
| f
|
|
0 |
0 |
0
45035996273736728 | node03
| /tmp/foo
| TEMP
| f
|
|
0 |
0 |
0
45035996273736730 | node04
| /tmp/foo
| TEMP
| f
|
|
0 |
0 |
0
(4 rows)
The RESTORE_LOCATION() function restores the location only on the nodes where the location
exists and is retired. The function does not propagate the storage location to nodes where that
location did not previously exist.
Restoring on all nodes fails if the location has been dropped on any of them. If you have dropped the
location on some nodes, you have two options:
l
If you no longer want to use the dropped node, restore the location individually on each of the
other nodes.
Alternatively, you can re-create the location on the node where you dropped it. To do so, use
CREATE LOCATION() or ADD_LOCATION(). After you have re-created the location, you can
then restore it on all nodes.
Administrator's Guide
Managing Storage Locations
The following example demonstrates the failure if you try to restore on nodes where you have
dropped the location:
=> select retire_location('/tmp/foo', '');
retire_location
-----------------------/tmp/foo retired.
(1 row)
=> select drop_location('/tmp/foo', 'node02');
drop_location
-----------------------/tmp/foo dropped.
(1 row)
=> select * from storage_locations where location_path ilike '/tmp/foo';
location_id
| node_name | location_path | location_usage | is_retired | location_label
| rank | throughput | latency
-------------------+-----------+---------------+----------------+------------+---------------+------+------------+--------45035996273736724 | node01
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
45035996273736728 | node03
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
45035996273736730 | node04
| /tmp/foo
| TEMP
| t
|
|
0 |
0 |
0
(3 rows)
=> select restore_location('/tmp/foo', '');
ERROR 2081: [/tmp/foo] is not a valid storage location on node node02
Administrator's Guide
Backing Up and Restoring the Database
Note: Saving a database backup on a different cluster does not provide Disaster Recovery.
The cloned database you create with vbr.py is entirely separate from the original, and is not
kept in sync with the database from which it is cloned.
Administrator's Guide
Backing Up and Restoring the Database
Types of Backups
The vbr.pyutility supports several kinds of backups:
l
Full backups
Object-level backups
You can refer to full or object-level backups by a user-defined descriptive name, such as
FullDBSnap, Schema1Bak, Table1Bak, and so on.
Note: This section uses backup as a noun (create a full backup), and back up as a multi-word
verb (be sure to back up the database).
Full Backups
A full backup is a complete copy of the database catalog, its schemas, tables, and other objects. It
is a consistent image, or snapshot, of the database at the time the backup occurred. You can use a
full backup for disaster recovery to restore a damaged or incomplete database.
Once a full backup exists, vbr.py creates incremental backups as successive snapshots following
the full-backup, consisting of new or changed data since the last full, or incremental, backup
occurred.
Archives consist of a collection of same-name backups. Each archive can have a different retention
policy. For example, if TBak is the name of an object-level backup of table T, and you create a daily
backup each week, the seven backups are part of the TBak archive. Keeping a backup archive lets
you revert back to any one of the saved backups.
Full and object-level backups reside on backup hosts, the computer system(s) on which backups
and archives are stored.
Backups are saved in a specific backup location, the directory on a backup host. This location can
comprise multiple backups, including associated archives. All backups in the same backup location
share data files (through hard links). The backups are also compatible, meaning that you can
restore any object-level backup from the same location, after restoring a full database backup.
Administrator's Guide
Backing Up and Restoring the Database
Object-Level Backups
An object-level backup consists of one or more schemas or tables, or a group of such objects. The
conglomerate parts of the object-level backup do not contain the entire database. Once an objectlevel backup exists, you can restore all of its contents, but not individually.
Note: HP Vertica does not support object level backups on Hadoop Distributed File System
(HDFS) storage.
Object level backups consist of the following object types:
Selective objects
Dependent objects
Principal objects
The objects on which both selected and dependent objects depend are
called principal objects. For example, each table and projection has an
owner, and each is a principal object.
Related Tasks
l
Administrator's Guide
Backing Up and Restoring the Database
If the epoch in the latest backup is earlier than the current ancient history mark.
Before and after you add, remove, or replace nodes in your database cluster.
Note: When you restore a full database backup, you must restore to a cluster that is identical
to the one on which you created the backup. For this reason, always create a new full backup
after adding, removing, or replacing nodes.
Ideally, create regular backups to back up your data. You can run the HP Vertica vbr.py from a
cron job or other task scheduler.
Related Terms
l
Administrator's Guide
Backing Up and Restoring the Database
Your backup directory, whether local or remote, must support hard links, even if you do not set
hardLinkLocal to True. Storage locations that do not support hard links, such as MapRNFS,
are incompatible with vbr.py.
Note: The version of rsync included in the HP Vertica installer supports a combined maximum
500 full- and object-level backups at one backup location. All backup hosts must use the rsync
version supplied with HP Vertica.
Administrator's Guide
Backing Up and Restoring the Database
If your site uses multiple backup host locations, you can estimate the database size requirements
per node with a query such as the following, substituting a backup host name for node_name:
select node_name,sum(used_bytes) as size_in_bytes from v_monitor.storage_containers group
by node_name;
Have an account on each backup host, with write permissions to the backup directory and to the
tempDir directory specified in the vbr.py configuration file. By default, this value is /tmp/vbr.
Have passwordless SSH access from each database cluster host to the corresponding backup
host
Administrator's Guide
Backing Up and Restoring the Database
How you fulfill these requirements depends on your platform and infrastructure.
SSHaccess among the backup hosts and access from the backup host to the database node is not
neccessary.
If your site does not use a centralized login system (such as LDAP), you can usually add a user
with the useradd command, or through a GUI administration tool. See the documentation for your
Linux distribution for details.
If your platform supports it, you can enable passwordless SSH logins using the ssh-copy-id
command to copy a database administrator's SSH identity file to the backup location from one of
your database nodes. For example, to copy the SSH identity file from a node to a backup host
named backup01:
> ssh-copy-id -i dbadmin@backup01|
Password:
Try logging into the machine with "ssh 'dbadmin@backup01'", and check the contents of the
~/.ssh/authorized_keysfile to make sure you have not added extra keys that you were not
expecting.
> ssh backup01
Last login: Mon May 23 11:44:23 2011 from host01
Repeat the steps to copy a database administrator's SSH identity to all backup hosts you will use
to back up your database.
After copying a database administrator's SSH identity, you should be able to log in to the backup
host from any of the nodes in the cluster without being prompted for a password.
Administrator's Guide
Backing Up and Restoring the Database
See Also
l
Related Terms
l
database administrator
Related Tasks
l
Administrator's Guide
Backing Up and Restoring the Database
Since you are creating a local backup, the host_name listed under the [Mapping] section in the
configuration file, after each node name is the name of the host on which you are creating the local
backup. For example, the following Mapping section shows mappings for a 3-node cluster. The
backupHost value you supplied when creating the configuration file is the name of the cluster host
(192.168.223.33) you are using as the hard link local backup host for the cluster:
[Mapping]
v_vmart_node0001 = 192.168.223.11:/home/dbadmin/data/backups
v_vmart_node0002 = 192.168.223.22:/home/dbadmin/data/backups
v_vmart_node0003 = 192.168.223.33:/home/dbadmin/data/backups
Administrator's Guide
Backing Up and Restoring the Database
The script prompts you to answer several questions, as shown in the following summarized
example:
[dbadmin@localhost ~]$/opt/vertica/bin/vbr.py --setupconfig
Snapshot name (backup_snapshot): fullbak1
Backup vertica configurations? (n) [y/n]: y
Number of restore points (1): 3
Specify objects (no default):
Vertica user name (dbadmin):
Save password to avoid runtime prompt? (n) [y/n]: y
Password to save in vbr config file (no default):
Node v_vmart_node0001
Backup host name (no default): 127.0.0.1
Backup directory (no default): /home/dbadmin/backups
Config file name (fullbak1.ini):
Password file name (no default value) (no default):pwdfile
Change advanced settings? (n) [y/n]: n
Saved vbr configuration to fullbak1.ini.
Saved vbr configuration to pwdfile.ini.
The following sections describe the information you supply.
Administrator's Guide
Backing Up and Restoring the Database
The configuration file for the object-level backup, called objectbak.ini, has these parameter
values:
snapshotName=objectbak
backupDir=/home/dbadmin/data/backups
Administrator's Guide
Backing Up and Restoring the Database
Note: This user name must be the user identified as the database administrator. No other user
has permission to invoke this action, even if the user has dbadmin permissions within the
database.
Administrator's Guide
Backing Up and Restoring the Database
Note: Because the backup configuration file is typically stored on the cluster you are backing
up, Hewlett-Packard recommends that you also save a copy of the configuration file on the
backup host. In this way, the configuration file remains available if the cluster node is lost.
Administrator's Guide
Backing Up and Restoring the Database
To continue without advanced settings and to save the configuration file with the information you
just entered, press Enter.
See Also
l
Related Tasks
l
Administrator's Guide
Backing Up and Restoring the Database
The utility prompts you to answer the questions listed in the following table. Each is followed by its
default value, if any exists. Type Enter to accept a default value.
The second table column lists the name of the parameter associated with the question, as it
appears in the configuration file.
snapshotName
verticaConfig
restorePointLimit
objects
dbUser
dbPromptForPassword
dbPassword
backupHost
backupDir
N/A
N/A
To change any advanced parameters, respond to the last question, Change advanced settings?
by entering y.
After you successfully complete all of the required questions, vbr.py generates a configuration file
with the information you supplied. Create separate configuration files for a full backup and for each
object-level backup. Use distinct backup names in each configuration file.
When the setup completes processing, enter a configuration file name. Use this file name when you
run the --task backup or other commands. The utility uses the configuration file contents for both
backup and restore tasks, and also for the --copycluster task.
If you do not successfully complete all of the required questions, vbr.py lists error messages and
hints, but does not create a configuration file. You can then run the --setupconfig command again
to specify any missing or incorrect information.
Administrator's Guide
Backing Up and Restoring the Database
See Also
l
Related Tasks
l
After you complete the required parameter options, continue to advanced settings. Respond to the
last question by entering y in response:
Change advanced settings? (n)[y/n]: y
Administrator's Guide
Backing Up and Restoring the Database
The completed configuration file includes the following list of the advanced parameters, their
description, and their default values:
Configuration
Parameter
bwlimit
checksum
When set to y, uses checksum for data integrity instead of file date and
size. The default value is n.
concurrency_backup
concurrency_restore
encrypt
port_rsync
The Port number for the rsync daemon. The default value is 50000.
retryCount
retryDelay
serviceAccessUser
tempDir
total_bwlimit_backup
total_bwlimit_restore
Administrator's Guide
Backing Up and Restoring the Database
See Also
l
Related Tasks
l
Administrator's Guide
Backing Up and Restoring the Database
hardLinkLocal = True
Administrator's Guide
Backing Up and Restoring the Database
snapshotName = exampleBackup
; For simplicity, use the same temp directory location on
; all backup hosts. The utility must be able to write to this
; directory.
tempDir = /tmp/vbr
; Vertica binary directory should be the location of
; vsql & bootstrap. By default it's /opt/vertica/bin
;verticaBinDir =
; include vertica configuration in the backup
verticaConfig = True
; how many times to rety operations if some error occurs.
retryCount = 5
retryDelay = 1
restorePointLimit = 5
[Database]
; db parameters
dbName = exampleDB
dbUser = dbadmin
dbPassword = password
; if this parameter is True, vbr will prompt user for db password every time
dbPromptForPassword = False
[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
; total bandwidth limit for all backup connections in KBPS, 0 for unlimited
total_bwlimit_backup = 0
; total number of backup connections, -u for unlimited
concurrency_backup = 0
; total bandwidth limit for all restore connections in KBPS, 0 for unlimited
total_bwlimit_restore = 0
; total number of restore connections, -u for unlimited
concurrency_restore = 0
[Mapping]
; backupDir ignored for copy cluster task
v_exampledb_node0001 = backup01:/home/dbadmin/backups
v_exampledb_node0002 = backup02:/home/dbadmin/backups
v_exampledb_node0003 = backup03:/home/dbadmin/backups
Related Tasks
l
Administrator's Guide
Speed Hard link local backups are significantly faster than a remote host backup. In a hard
link local backup, vbr.py does not copy files (as long as the backup directory exists on the
same file system as the database catalog and data directories).
Reduced network activities The hard link local backup minimizes network load because it
does not require rsync to copy files to a remote backup host.
Less disk space Since the backup includes a copy of the catalog and hard file links, the local
backup uses significantly less disk space than a backup with copies of database data files.
However, since a hard link local backup saves a full copy of the catalog each time you run
vbr.py, the disk size will increase with the catalog size over time.
Hard link local backups are useful during experimental designs and development cycles. Database
designers and developers can create hard link local object backups of schemas and tables on a
regular schedule during design and development phases. If any new developments are
unsuccessful, developers can restore one or more objects easily and efficiently.
Note: Running vbr.py does not affect active database applications. The vbr.py utility
supports creating backups while concurrently running applications executing DML statements,
including COPY, INSERT, UPDATE, DELETE, and SELECT.
Administrator's Guide
system as the database data and catalog files. The Linux operating system cannot create hard file
links to another file system.
Do not create the hard link local backup directory in a database data storage location. For example,
as a best practice, the database data directory should not be at the top level of the file system, such
as this:
/home/dbadmin/data/VMart/v_vmart_node0001
Instead, HP Vertica recommends adding another subdirectory for data above the database level,
such as in this example:
/home/dbadmin/data/dbdata/VMart/v_vmart_node0001
You can then create the hard link local backups subdirectory as a peer of the data directory you just
created, such as in this example:
/home/dbadmin/data/backups
/home/dbadmin/data/dbdata
For more information about how vbr.py creates directories for backups and archives, see Backup
Directory Structure and Contents in the See Also section below.
See Also
l
Administrator's Guide
Related Tasks
l
The backup host (either on the database cluster or not) has sufficient disk space to store the
backups
The user account of whoever starts the utility (dbadmin, or other) has write access to the target
directories on the host backup location.
Run the vbr.py script from a terminal using the database administrator account from an initiator
node in your database cluster. You cannot run the utility as root.
Note: If you have upgraded from HP Vertica 5.x to 6.x, backups created with 5.x are
incompatible with 6.x. Once you have created new backups with the 6.x vbr.py utility,
consider removing older backups created with 5.x.
--task backup
--config-file config_file
If your configuration file does not contain the database administrator password, vbr.py prompts
you to enter the password, but does not display what you type:
Administrator's Guide
By default, there is no screen output, other than the progress bar. To include additional
progress information, use the --debug option, with a value between 1 3.
2. If the utility does not locate the configuration you specify, it fails with an error and exits.
Use the same backup host directory location for both kinds of backups
For best network performance, use one backup host per cluster node
For future reference, append the major Vertica version number to the configuration file name
(mybackup76x)
Using the same backup host directory location for full- and object-level backups results in the
backups sharing disk space and being compatible when performing a restore. Each cluster node
must also use the same directory location on its designated backup host.
Administrator's Guide
The selected objects of a backup can include one or more schemas or tables, or a combination of
both. For example, you can include schema S1 and tables T1 and T2 in an object-level backup.
Multiple backups can be combined into a single backup. A schema-level backup can be integrated
with a database backup (and a table backup integrated with a schema-level backup, and so on).
Object-Level Backups
After you create a full backup, you can create object-level backups.
To see existing full and incremental backups, see Viewing and Removing Backups in the See Also
section below.
The value of the snapshotName parameter in the configuration file. This is the name used as a prefix to
the container directory on the backup host.
Administrator's Guide
_
archi
ve
date_
timest
amp
The prefix (_archive) indicates the directory contains an archive. The vbr.py utility appends the date, in
the form date_timestamp to the directory name when the backup is created. For instance, if you specify
the backup name as mybak, and create backups over three consecutive days, multiple subdirectories with
archive names such as these will exist:
l
mybak_archive20111111_205841
mybak_archive20111112_205844
mybak_archive20111113_206841
See Also
l
Viewing Backups.
Administrator's Guide
Related Terms
l
K-safe
database administrator
Related Tasks
l
Directory Tree
The directory structure starts at the location you specify as the backupDir parameter value. In that
directory, vbr.py creates the top-level backup subdirectory using the snapshotName value in the
configuration file. The default configuration file name consists of the snapshotName value, with an
.ini suffix. You can use any valid file name for the configuration file, or change the name once the
file exists. Throughout this example, the backup name is objectbak, and the configuration file
name is objectbak.ini.
l
The top level directory starts with the backup name you specified in the snapshotName
parameter value, and creates a structure based on the cluster nodes listed in the configuration
file. In the following example, the configuration file parameter snapshotName=objectbak, and
the three dbNode parameters specified are v_vmart_node0001, v_vmart_node0002, and v_
vmart_node0003:
Administrator's Guide
When you create a backup, for each designated node, vbr.py creates a subdirectory below the
node directory, also named by the snapshotName parameter value for the backup name:
If the value is greater than 1, the utility checks to see how many backups and archives exist in
each node's backup name subdirectory. If no archive files exist, the utility creates the first
backup in a new subdirectory with the backup name, one per node.
The backup name subdirectory for each node also contains a home subdirectory that includes
two important files, backupname.info and backupname.txt. Among other information, these
files contain the archive OID of each backup (unique across all nodes), and the backup epoch:
Administrator's Guide
The next time you create a backup using the same configuration file, the restorePointLimit
value is verified again.If vbr.py is to save another backup, it gets the OID from the info file of
the existing backup, and saves that previous backup to a new directory. The new directory for
the last backup is then an archive, named with the backup name (objectbak in this example),
and the OID suffix _archivedatestamp_timestamp.
The following example lists the backup directory, objectbak, with subdirectories for each node.
Each node then has one subdirectory for the backup (objectbak) and a peer subdirectory for the
first archive:
Thereafter, each time you create a backup, the utility continues this process until the number of
backups and archives together equal the restorePointLimit value. Then, vbr.py deletes the
oldest archive backup, and saves a new backup.
Once you save multiple backups, you can restore directly from the archive of your choice.
Related Tasks
l
Administrator's Guide
Object
Overwrite
For more information about creating configuration files for full- or object-level backups, see
Configuring Required VBRParameters in the Related Tasks section below. Configuring Required
VBR Parameters
Administrator's Guide
For best network performance, use one backup host per cluster node
For future reference, append the major Vertica version number to the configuration file name
(mybackup7x)
Using the same backup host directory location for full- and object-level backups results in the
backups sharing disk space and being compatible when performing a restore. Each cluster node
must also use the same directory location on its designated backup host.
The selected objects of a backup can include one or more schemas or tables, or a combination of
both. For example, you can include schema S1 and tables T1 and T2 in an object-level backup.
Multiple backups can be combined into a single backup. A schema-level backup can be integrated
with a database backup (and a table backup integrated with a schema-level backup, and so on).
Naming Conventions
Give each object-level backup configuration file a distinct and descriptive name. For instance, at
the airport terminal, schema-based backup configuration files use a naming convention with an
airline prefix, followed by further description, such as:
Administrator's Guide
AIR1_daily_arrivals_snapshot
AIR2_hourly_arrivals_snapshot
AIR2_hourly_departures_snapshot
AIR3_daily_departures_snapshot
Once database and object-based backups exist, you can recover the backup of your choice.
Note: Do not change object names in an object-level configuration file once a backup already
exists. When the backup name and backup locations are the same, even if you change the
objects listed in the configuration file, the later backup overwrites the first so you cannot
restore from the earlier backup. Instead, create a different configuration file.
Give each object backup initiated on a different node a unique backup name.
Start the backup script on different initiator nodes to create their specific tenant backups
concurrently.
Administrator's Guide
restore the schema or table, including the selected, dependent, and principal objects. In summary,
an object-level backup includes the following contents:
l
Catalog snippet: persistent catalog objects serialized into the principal and dependent objects
Some of the elements that comprise AIR2, for instance, include its parent schema, tables, named
sequences, primary key and foreign key constraints, and so on. To create such a backup, vbr.py
script saves the objects directly associated with the table, along with any dependencies, such as
foreign key (FK) tables, and creates an object map from which to restore the backup.
Note: Because the data in local temp tables persists only within a session, local temporary
tables are excluded when you create an object-level schema. For global temporary tables,
vbr.py stores the table's definition.
Administrator's Guide
Administrator's Guide
Identity and auto-increment sequences are dependent objects, since they cannot exist without their
tables. Such objects are backed up along with the tables on which they depend.
Named sequences are not dependent objects, since they exist autonomously. A named sequence
remains after dropping the table in which the sequence is used. In this case, the named sequence is
a principal object that must be backed up with the table and regenerated if it does not already exist
when you restore the table. If the sequence does exist, it is used, unmodified. Sequence values
could repeat, if you restore the full database and then restore a table backup to a newer epoch.
Administrator's Guide
Backup Epochs
Each backup includes the epoch to which its contents can be restored. This epoch will be close to
the latest epoch, though the epoch could change during the time the backup is being written. If an
epoch change occurs while a backup is being written, the storage is split to indicate the different
epochs.
The vbr.py utility attempts to create an object-level backup five times before an error occurs and
the backup fails.
See Also
l
Viewing Backups
Types of Backups
Related Tasks
l
Administrator's Guide
Your database is running. All nodes need not be up in a K-safe database for vbr.py to run.
However, keep in mind that any nodes that are down are not backed up.
The user account of whoever starts the utility (dbadmin, or other) has write access to the target
backup directories.
When you create a full- or object-level hard link local backup, these are the backup contents:
Backup
Full backup
Full
copy
Object-level
Full
Hard file links for all objects listed in the configuration file, and any of
backup
copy
Run the vbr.py script from a terminal using the database administrator account from a node in your
database cluster. You cannot run the utility as root.
To create a full or object-level backup, enter the following command:
> /opt/vertica/bin/vbr.py --task backup --config fullbak.ini
Note: While not required, HP Vertica recommends that you first create a full backup before
creating any object-level backups.
Administrator's Guide
database files.
The following steps present a simplified approach to saving, and then restoring, hard link local
backups from tape storage:
1. Create a configuration file using a command such as this:
/opt/vertica/bin/vbr.py --setupconfig
2. Edit the configuration file (localbak.ini in this example) to include the hardLinkLocal=True
parameter in the [Transmission] section.
3. Run the backup utility with the configuration file:
/opt/vertica/bin/vbr.py --task backup --config-file localbak.ini
4. Copy the hard link local backup directory with a separate process (not vbr.py) to tape or other
external media.
5. If the database becomes corrupted, create the directory structure that existed when you
created the hard link local backup.
6. Transfer the backup files from tape to their original backup directory.
7. Using the configuration file you used to create the hard link local backup (Step 3), restore the
database using the following command:
/opt/vertica/bin/vbr.py --task restore --config-file localbak.ini
When you restore from a hard link local backup (copied from tape), vbr.py creates hard links
from the backup files to the database directory, if possible, again saving significant disk space,
and time.
Related Terms
l
K-safe
database administrator
Administrator's Guide
Viewing Backups
These are the ways to view backups:
1. Using vbr.py, list the backups that reside on the local or remote backup host (requires a
configuration file).
2. Monitor backup information while vbr.py is executing. See the DATABASE_SNAHPSHOTS
description in the Related Terms section below.
3. To view historical information about backups, see the DATABASE_BACKUPS description in
the Related Terms section below.
Note: Since the database_backups system table contains historical information, it is not
updated when you delete the backups.
Administrator's Guide
/home/dbadmin/table2bak.ini
Found 1 backup(s) with given config file /home/dbadmin/table2bak.ini.
backup epoch objects hosts(nodes) file_system_type
table2bak_20130516_072002 180 test2 v_vmart_node0001(192.168.223.33),
v_vmart_node0002(192.168.223.33), v_vmart_node0003(192.168.223.33) [Linux]
The vbr.py output information includes the backup OID, the epoch, which object(s) (if applicable)
were backed up, and the backup host (192.168.223.33) used for each node. The identifier
appended to the backup name (20130516_072002 in this example), is the value you need when
specifying an archive to restore.
Monitor database_snapshots
You can monitor the backups while vbr.py is in process. The following examples show the system
table while a single host node is being backed up:
VMart=> select * from database_snapshots;
node_name
| snapshot_name | is_durable_snapshot | total_size_bytes | storage_cost_
bytes | acquisition_timestamp
-----------------+---------------+---------------------+------------------+-------------------+-----------------------v_vmart_node0001 | fullbak
| t
|
299089541 |
863439 | 2013-09-04 14:15:23-07
(1 row)
Query database_backups
Use the following query to list historical information about backups. The objects column lists
which objects were backed up in object-level backups. However, do not use the backup_
timestamp value when restoring an archive. Use the values in the vbr.py --task listbackup,
described in the previous section when restoring an archive.
VMart=> select * from v_monitor.database_backups;
-[RECORD 1 ]----+-----------------------------backup_timestamp | 2013-05-10 14:41:12.673381-04
node_name
| v_vmart_node0003
snapshot_name
| schemabak
backup_epoch
| 174
node_count
| 3
file_system_type | [Linux]
objects
| public, store, online_sales
-[RECORD 2 ]----+-----------------------------backup_timestamp | 2013-05-13 11:17:30.913176-04
node_name
| v_vmart_node0003
snapshot_name
| kantibak
backup_epoch
| 175
node_count
| 3
Administrator's Guide
file_system_type | [Linux]
objects
|
-[RECORD 13 ]---+-----------------------------backup_timestamp | 2013-05-16 07:02:23.721657-04
node_name
| v_vmart_node0003
snapshot_name
| objectbak
backup_epoch
| 180
node_count
| 3
file_system_type | [Linux]
objects
| test, test2
-[RECORD 14 ]---+-----------------------------backup_timestamp | 2013-05-16 07:19:44.952884-04
node_name
| v_vmart_node0003
snapshot_name
| table1bak
backup_epoch
| 180
node_count
| 3
file_system_type | [Linux]
objects
| test
-[RECORD 15 ]---+-----------------------------backup_timestamp | 2013-05-16 07:20:18.585076-04
node_name
| v_vmart_node0003
snapshot_name
| table2bak
backup_epoch
| 180
node_count
| 3
file_system_type | [Linux]
objects
| test2
Related Terms
l
DATABASE_SNAPSHOTS
DATABASE_BACKUPS
Related Tasks
l
Removing Backups
The database is down. You cannot restore a full backup when the database is running.
The backup directory exists and contains the backups from which to restore.
Administrator's Guide
The cluster to which you are restoring the backup has the same number of hosts as the one used
to create the backup. The node names and the IP addresses must also be identical.
The database you are restoring must already exist on the cluster to which you are restoring data.
The database can be completely empty without any data or schema. As long as the database
name matches the name in the backup, and all of the node names match the names of the nodes
in the configuration file, you can restore to it.
You can use only a full database backup to restore a database from scratch. If you have saved
multiple backup archives, you can restore from either the last backup, or a specific archive.
When you restore a full database backup, OIDs from each backup (including all archives) in the
same backup are injected into the restored catalog of the full database backup. Additionally, the
OID generator and current epoch are set to the respective maximum OID and epoch from the full
database backup.
You can restore a snapshot only to the database from which it was taken. You cannot restore a
snapshot into an empty database.
Administrator's Guide
Restoring an Archive
If you saved multiple backups, you can specify a specific archive to restore. To list the archives
that exist to choose one to restore, use the vbr.py --listbackup task, with a specific
configuration file. For more information, see Viewing and Removing Backups in the See Also
section below.
To restore from one of several archives:
1. Log in using the database administrator's account. You cannot run the utility as root.
Invoke the utility with the --task restore command, the configuration file with which you
created the backup, followed by the --archive parameter with the date_timestamp suffix of
the directory name to identify which archive to restore. For example:
> vbr.py --task restore --config-file fullbak.ini --archive=20121111_205841
To restore the node, first bring it down, and then run the utility again.
Administrator's Guide
Attempting to restore a full backup to an alternate cluster when the source database is accessible
can render the source database unusable. The backup you are restoring contains critical references
to the source database and uses that information during the restore process.
See Also
l
Monitoring Recovery
Viewing Backups
Related Terms
l
database administrator
Backup Locations
Backups created in the same backup location are compatible. Be sure, however, to assign different
file names to each backup. Using the same
After restoring a full database backup, you can restore object-level backups saved in the same
backup location as the full backup.
Note: Using different backup locations in which to create full- or object-level backups results in
incompatible object-level backups. Attempting to restore an object-level backup after restoring
a full database will fail.
Administrator's Guide
Restoring an object-level backup does not affect any schemas or tables that were not included in
the original backup you are restoring. For instance, in a multi-tenanted database, if you create
schema backups for each of four different tenants, restoring one tenant schema does not overwrite
the other three.
Administrator's Guide
It must recover any data that it does not have (by comparing epochs and other recovery
procedures)
Administrator's Guide
event occurred. If a node misses the restore table event, it will recover projections anchored on the
given table.
See Also
l
Failure Recovery
Transactions
Related Terms
l
LOCK_USAGE
Related Tasks
l
Administrator's Guide
2. Create an object-level hard link local backup for Table1 using the same backup directory, with
configuration file table1bak.ini:
[dbadmin@node01 ~]$/opt/vertica/bin/vbr.py --task backup --config-file table1bak.ini
3. Create an object-level hard link local backup for Table2, using the same backup directory, with
configuration file table2bak.ini.
[dbadmin@node01 ~]$/opt/vertica/bin/vbr.py --task backup --config-file table2bak.ini
After creating these hard link local backups, using the /backups directory, the following directory
structure exists:
Given this situation, the following occurs when you need to restore from a hard link local backup:
l
When restoring a full DB (when table1 and table2 backups exist in the same backup
directory), vbr.py detects the maximum object ID (OID) and epochs from the object-level table
backups and sets them in the restored database. This prevents OID and epoch conflicts from
Administrator's Guide
occurring when object-level backups are restored after the newly restored database.
l
If the database and object-level table backups are not in the same backup directory, vbr.py
reverts the maximum OID and epoch for a full database restore back to the table OID and epoch.
Further attempts to restore either table1 or table2 backups will then fail, preventing any
potential conflicts.
2. Copy the backup files to their original backup directory, as specified for each node in the
configuration file with the backupHost and backupDir parameters. For example, this
configuration file shows the backupDir parameter for v_vmart_node0001:
[Mapping0]dbNode = v_vmart_node0001
backupHost = node03
backupDir = /home/dbadmin/backups/localbak
3. To restore the latest version of the backup, move the backup files to this directory:
/home/dbadmin/backups/localbak/node_name/snapshotname
4. To restore a different backup version, move the backup files to this directory:
/home/dbadmin/backups/localbak/node_name/snapshotname_archivedate_timestamp
5. When the backup files are returned to their original backup directory, use the original
Administrator's Guide
If the physical files are restored from tape into the correct directories, and you use the
configuration file that specifies hardLinkLocal = true, restoring the backup succeeds.
Note: You can use a different directory when you return the backup files to the hard link local
backup host. However, you must also change the backupDir parameter value in the
configuration file before restoring the backup.
Note: If you restore data to a single node, the node has already stopped. You do not need
to stop the database.
Note: If you restored all the nodes using the backup, you are using a manual recovery
method. For more information, see the Failure Recovery in the See Also section below.
Administration Tools returns the message, "Database startup failed," after you attempt to
restart the database and then offers to restart the database from an earlier epoch. Click
Yes.
4. After the database starts, connect to it through the Administration Tools or MC and verify that it
was successfully restored by running some queries.
See Also
l
Failure Recovery
Administrator's Guide
Related Tasks
l
Removing Backups
To remove existing backups:
1. Use vbr.py --task listbackup to identify the backup and archive names that reside on the
local or remote backup host (requires a configuration file).
2. Delete the backup directories.
snapshotName
backupHost
backupDir
2. Also note the dbNode parameter values, indicating which nodes were backed up.
3. Connect to the backup host.
4. Navigate to the backup directory (the backupDir parameter value in the configuration file).
Administrator's Guide
Administrator's Guide
storage_usage | DATA
-[RECORD 5 ]-+----------------------------------------------------node_name
| v_vmart_node0001
storage_path | /home/dbadmin/SSD/schemas
storage_usage | DATA
-[RECORD 6 ]-+----------------------------------------------------node_name
| v_vmart_node0002
storage_path | /home/dbadmin/VMart/v_vmart_node0002_catalog/Catalog
storage_usage | CATALOG
-[RECORD 7 ]-+----------------------------------------------------node_name
| v_vmart_node0002
storage_path | /home/dbadmin/VMart/v_vmart_node0002_data
storage_usage | DATA,TEMP
-[RECORD 8 ]-+----------------------------------------------------node_name
| v_vmart_node0002
storage_path | /home/dbadmin/SSD/tables
storage_usage | DATA
.
.
.
Notice the directory paths for the Catalog, Data, and Tempstorage. These paths are the same on all
nodes in the source database, and must be the same in the target database.
Note: When copying a database to another cluster, if the target data is different, vbr.py
overwrites all existing data. If the target data is identical, vbr.py does nothing. To retain
existing data on the target cluster, create a full database backup of the target before invoking
the copycluster vbr.py task.
-orTo run Admintoolsfrom the command line, enter a command such as this for the VMart database:
$/opt/vertica/bin/admintools -t node_map -d VMART
Administrator's Guide
DATABASE | NODENAME
| HOSTNAME
----------------------------------------------VMART
| v_vmart_node0001 | 192.168.223.xx
VMART
| v_vmart_node0002 | 192.168.223.yy
VMART
| v_vmart_node0003 | 192.168.223.zz
Have a database with the same name as the source database. The target database can be
completely empty.
Have the same node names as the source cluster. The nodes names listed in the NODES
system tables on both clusters must match.
Have the same database administrator account, and all nodes must allow a database
administrator of the source cluster to login through SSH without a password.
Note: Having passwordless access within the cluster is not the same as having
passwordless access between clusters. The SSH ID of the administrator account on the
source cluster is likely not the same as the SSH ID of the administrator account on the
target cluster. You need to configure each host in the target cluster to accept the SSH
authentication of the source cluster.
Have adequate disk space for the vbr.py --task copycluster command to complete.
Administrator's Guide
The configuration file in the following example is set up to copy a database on a three node cluster
(v_vmart_node0001, v_vmart_node0002, and v_vmart_node0003) to another cluster consisting of
nodes named test-host01, test-host02, and test-host03.
[Misc]
snapshotName = CopyVmart
verticaConfig = False
restorePointLimit = 5
tempDir = /tmp/vbr
retryCount = 5
retryDelay = 1
[Database]
dbName = vmart
dbUser = dbadmin
dbPassword = password
dbPromptForPassword = False
[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
[Mapping]
; backupDir is not used for cluster copy
v_vmart_node0001= test-host01:/home/dbadmin/backups
v_vmart_node0002= test-host02:/home/dbadmin/backups
v_vmart_node0003= test-host03:/home/dbadmin/backups
Administrator's Guide
Related Terms
l
Database Administrator
NODES
Related Tasks
l
Administrator's Guide
Syntax
/opt/vertica/bin/vbr.py {command }
... [--archive file ]
... [--config-file file ]
... [--debug <em+>level</em>]
... [--nodes node1
[,
noden, ...] ]
... [--showconfig ]
Short Command
Description
--help
-h
--setupconfig
Administrator's Guide
Full Command
Short Command
Description
--task {backup
| copycluster
| listbackup
| restore }
-t
Parameters
Parameter
Description
--archive file
--config-file file
Indicates the configuration file to use. If this file does not exist, an error
occurs and the utility cannot continue. The file parameter can be absolute
or relative path to the location from which you start the backup utility.
Administrator's Guide
Parameter
Description
--nodes node1[,...]
Specifies the node or nodes (in a comma-separated list if more than one
node exists), on which to perform a vbr.py task. The nodes in the list are
the same names in the Mapping section of the configuration file. See the
VBRConfiguration File Reference in the See Also section belowfor
details.
Caution: If you create a backup for only some nodes in the cluster, you
are creating a partial database backup. Partial backups can result in lost
data if not used correctly. Do not try to restore the entire database cluster
from a partial database backup created from a subset of the nodes.
--debug level
--showconfig
The configuration values being used to perform the task. The parameters
are shown in a raw JSON format before vbr.py starts processing.
See Also
l
Related Tasks
l
Administrator's Guide
Parameter
Default
snapshotName
snapshotName
tempDir
/tmp/vbr
az
AZ
09
hyphen (-)
underscore (_)
verticaBinDir
/opt/vertica/bin
verticaConfig
False
Administrator's Guide
Parameter
Default
restorePointLimit
objects
None
Administrator's Guide
Parameter
Default
overwrite
True
retryCount
retryDelay
Default Description
dbName
N/A
Administrator's Guide
Parameter
Default Description
dbUser
Current
Identifies the login name of the person who can run vbr.py to
user
name
dbPromptForPassword True
Default Description
encrypt
False
Administrator's Guide
Parameter
Default Description
checksum
False
Controls whether the vbr.py utility has rsync use the md5
checksum to determine whether files are identical before and
after network transmission. By default, rsync does not perform
checksum. Instead, it performs minimal file checking,
confirming that the file size and time of last modification are
identical before and after transmission.
Note: Calculating checksum values increases processor usage
during the backup process. For more details, see the Wikipedia
rsync page in the Related Terms section below.
port_rsync
50000
Changes the default port number for the rsync protocol. Change
this value if the default rsync port is in use on your cluster, or
you need rsync to use another port to avoid a firewall restriction.
serviceAccessUser None
The user account for your rsync daemon. If you do not provide
an account, HP Vertica leaves rsync running without
authentication, creating a potential security risk. If you choose
to save the password, HP Vertica stores it in the Password
Configuration File.
hardLinkLocal
False
port_ssh_backup
22
Overrides the default SSH port setting (22) for the backup
hosts. Enter the required SSH port for your site.
Changing the default SSH port is supported only when using
the backup and restore tasks. Using a non-default SSH port
with the copycluster task is not supported.
NOTE: This parameter is not included in the configuration file
automatically. See Configuring Backup Hosts in the Related
Tasks section below for details to enter the parameter
manually.
Administrator's Guide
[Mapping]
There is one [Mapping] section for all of the nodes in your database cluster. The section must exist
in your configuration file, since it specifies all database nodes being included in the backup, along
with the backup host and directory for each node.
NOTE:Previous configuration files used separate mapping sections for each database node.
While vbr.py continues to support that format, you cannot combine formats. If you edit an
existing configuration file to add a Mapping in the current style, you must combine information
from all existing Mappings into the new section. Alternatively, you can use vbr.py with the -task=create-config option to generate a new configuration file, as described in Creating
vbr.py Files, found in the Related Tasks section below.
In the following example, the Mapping section indicates a single node to be backed up (v_vmart_
node0001). The node is assigned to the backup host (127.0.0.1), and the backup directory
(/home/dbadmin/backups).Notice that, even though this example is for a single node cluster, and
the backup host and the database node are the same system, you specify them differently. The
backup host and directory specifications use a colon (:) as a separator:
[Mapping]
v_vmart_node0001 = 127.0.0.1:/home/dbadmin/backups
While the configuration file [Mapping] section no longer uses named parameters, the elements of
the simplified format continue to represent the following parameters:
dbNode = backupHost:backupDir
Parameter
Default Description
backupHost None
Administrator's Guide
Parameter
Default Description
backupDir
None
Identifies the full path to the directory on the backup host or node where
the backup will be stored.
Directory requirements:
l
Must already exist when you run the utility with the --task backup
option
Must be writable by the user account used to run the backup utility.
This setting is not used for the copycluster command. The backup
directory, whether local or remote, must support hard links, even if you
do not set hardLinkLocal to True.
dbNode
None
Related Terms
l
database administrator
Related Tasks
l
Administrator's Guide
HP Vertica creates the password file even if you do not save any passwords. In this case, the file
exists but remains empty.
Default Description
dbPassword
None
serviceAccessPass None
Administrator's Guide
Administrator's Guide
Recovering the Database
You can monitor a recovery in progress by viewing log messages posted to the vertica.log file on
each host.
See Also
l
Failure Recovery
Failure Recovery
Recovery is the process of restoring the database to a fully functional state after one or more nodes
in the system has experienced a software- or hardware-related failure. HP Vertica recovers nodes
by querying replicas of the data stored on other nodes. For example, a hardware failure can cause a
node to lose database objects or to miss changes made to the database (INSERTs, UPDATEs,
and so on) while offline. When the node comes back online, it recovers lost objects and catches up
with changes by querying the other nodes.
K-safety is a measure of fault tolerance in the database cluster. The value K represents the number
of replicas of the data in the database that exist in the database cluster. These replicas allow other
nodes to take over for failed nodes, allowing the database to continue running while still ensuring
data integrity. If more than K nodes in the database fail, some of the data in the database may
become unavailable. In that case, the database is considered unsafe and automatically shuts
down.
Administrator's Guide
Recovering the Database
It is possible for an HP Vertica database to have more than K nodes fail and still continue running
safely, because the database continues to run as long as every data segment is available on at
least one functioning cluster node. Potentially, up to half the nodes in a database with a K-safety
level of 1 could fail without causing the database to shut down. As long as the data on each failed
node is available from another active node, the database continues to run.
Note: If half or more of the nodes in the database cluster fail, the database will automatically
shut down even if all of the data in the database is technically available from replicas. This
behavior prevents issues due to network partitioning.
In HP Vertica, the value of K can be zero (0), one (1), or two (2). If a database that has a K-safety of
one (K=1) loses a node, the database continues to run normally. Potentially, the database could
continue running if additional nodes fail, as long as at least one other node in the cluster has a copy
of the failed node's data. Increasing K-safety to 2 ensures that HP Vertica can run normally if any
two nodes fail. When the failed node or nodes return and successfully recover, they can participate
in database operations again.
The physical schema design must meet certain requirements. To create designs that are K-safe,
HP recommends using the Database Designer.
Note: : You can monitor the cluster state through the View Database Cluster state menu
option.
Recovery Scenarios
Recovery comes into play when a node or the database is started. Depending upon how the node or
database was shut down, and how it is restored, there are three possibilities for a K-Safe database:
l
Recovery of failed nodes: One or more nodes have failed, but the database continues to run
since the remaining nodes in the database are able to fill in for the failed nodes. The failed nodes
can be restarted through the Administration Tools using the Restart HP Vertica on host option.
The nodes being restarted have a RECOVERING status while they rebuild some of the data
from the remaining nodes. Once rebuilding is finished, the nodes transition to an UP status. The
database can continue to commit transactions during the recovery process, except for a short
period at the end of the recovery process.
Recovery after a Clean Shutdown: The database had been shut down cleanly via the
Administration Tools Stop Database option. In this case, the database should be restarted using
the Start Database option. Upon restart all nodes that were 'UP' at the time of shutdown
immediately transition to 'UP'. It is possible that at the time of shutdown, the database had one
Administrator's Guide
Recovering the Database
or more failed nodes. If these nodes are now available, they go through the 'RECOVERING'
state as described in 'Recovery of failed nodes" case above.
l
Recovery after an Unclean Shutdown (Manual Recovery): The database was not shut down
cleanly, which means that the database became unsafe due to a failure. In this case, the
database possibly did not write all the data from the WOS to disk. There are several reasons for
unclean shutdowns, such as:
n
A site-wide event, such as a power failure that causes all nodes to reboot.
When the database is started through the Administration Tools Start Database option, recovery
determines that a normal startup is not possible. It goes on to determine a point in time in which
the data was consistent on all nodes. This is called the Last Good Epoch. As part of Start
Database processing, the administrator is prompted to accept recovery with the suggested
epoch. If accepted, the database recovers and any data changes made after the Last Good
Epoch are lost. If not accepted, startup is aborted and the database is not started on any of the
nodes.
Instead of accepting the given epoch, the administrator can instead choose to recover from a
backup or select an epoch for an even earlier point using the Roll Back Database to Last Good
Epoch option in the Administration Tools Advanced Menu. This is useful in special situations, for
example if the failure occurs during a batch of loads, for which it is easier to go back to the
beginning of the batch, rather than starting in the middle, even though some of the work must be
repeated. In most scenarios, it is sufficient and recommended to accept the given epoch.
Notes
l
In HP Vertica 5.0, manual recovery is possible as long as the nodes that are being started can
supply all of the partition segments in the database. This means that more than K nodes can
remain down at startup, and the database can still successfully start as long as all of the data is
available from the remaining nodes in the cluster.
In HP Vertica 4.1, the default for the HistoryRetentionTime configuration parameter changed
to 0, which means that HP Vertica only keeps historical data when nodes are down. This default
setting effectively prevents the use of the Administration Tools 'Roll Back Database to Last
Administrator's Guide
Recovering the Database
Good Epoch' option because the AHM remains close to the current epoch and a rollback is not
permitted to an epoch prior to the AHM. If you rely on the Roll Back option to remove recently
loaded data, consider setting a day-wide window for removing loaded data; for example:
=> ALTERDATABASE mydb SET HistoryRetentionTime = 86400;
Starting in 4.0, manual recovery is possible even if up to K nodes are out of commission; for
example, physically removed for repair or not reachable at the time of recovery. Once the nodes
are back in commission, they recover and rejoin the cluster, as described in the "Recovery after
failure of up to K nodes" section above.
IMPORTANT: When a node is down, it can take a full minute or more for the HP Vertica
processes to time out during its attempt to form a cluster when manual recovery is needed. Wait
approximately one minute until the system returns the manual recovery prompt. Do not press
CTRL-C during database startup.
See Also
l
Note: You might see additional nodes in the list, which are used internally by the
Administrator's Guide
Recovering the Database
4. Verify recovery state by selecting View Database Cluster State from the Main Menu.
After the database is fully recovered, you can check the status at any time by selecting View
Database Cluster State from the Administration Tools Main Menu.
Administrator's Guide
Recovering the Database
2. Start the database by selecting Start Database from the Main Menu.
3. Select the database you want to restart and click OK.
If you are starting the database after an unclean shutdown, messages display, which indicate
that the startup failed. Press RETURN to continue with the recovery process.
An epoch represents committed changes to the data stored in a database between two
specific points in time. When starting the database, HP Vertica searches for last good epoch.
4. Upon determining the last good epoch, you are prompted to verify that you want to start the
database from the good epoch date. Select Yes to continue with the recovery.
Administrator's Guide
Recovering the Database
Caution: If you do not want to start from the last good epoch, you may instead restore the
data from a backup and attempt to restart the database. For this to be useful, the backup
must be more current than the last good epoch.
HP Vertica continues to initialize and recover all data prior to the last good epoch.
If recovery takes more than a minute, you are prompted to answer <Yes> or <No> to "Do you
want to continue waiting?"
When all the nodes' status have changed to RECOVERING or UP, selecting <No> lets you
exit this screen and monitor progress via the Administration Tools Main Menu. Selecting
<Yes> continues to display the database recovery window.
Note: Be sure to reload any data that was added after the last good epoch date to which
Administrator's Guide
Recovering the Database
Monitoring Recovery
There are several ways to monitor database recovery:
l
System tables
Administrator's Guide
Recovering the Database
RECOVERY_STATUS
PROJECTION_RECOVERIES
Specifically, the recovery_status system table includes information about the node that is
recovering, the epoch being recovered, the current recovery phase, and running status:
=>select node_name, recover_epoch, recovery_phase, current_completed, is_running from
recovery_status;
node_name | recover_epoch | recovery_phase
| current_completed | is_running
-----------+---------------+-------------------+-------------------+-------------node01
|
|
| 0
| f
node02
| 0
| historical pass 1 | 0
| t
node03
| 1
| current
| 0
| f
To see a single record from the projection_recoveries system table, add limit 1 to the query.
Administrator's Guide
Recovering the Database
Note: You can also monitor the state of your database nodes on the Management Console
Overview page under the Database section, which tells you the number of nodes that are up,
critical, recovering, or down. To get node-specific information, click Manage at the bottom of
the page.
See Also
l
Monitoring HP Vertica
Exporting a Catalog
When you export a catalog you can quickly move a catalog to another cluster. Exporting a catalog
transfers schemas, tables, constraints, projections, and views. System tables are not exported.
Exporting catalogs can also be useful for support purposes.
See the EXPORT_CATALOG function in the SQL Reference Manual for details.
Recovery point objective (RPO): How much data loss can your organization tolerate upon a
disaster recovery?
Recovery time objective (RTO): How quickly do you need to recover the database following a
disaster?
Depending on your RPO and RTO, HP Vertica recommends choosing from the following solutions:
Administrator's Guide
Recovering the Database
1. Dual-load: During each load process for the database, simultaneously load a second
database. You can achieve this easily with off-the-shelf ETL software.
2. Periodic Incremental Backups: Use the procedure described in Copying the Database to
Another Cluster to periodically copy the data to the target database. Remember that the script
copies only files that have changed.
3. Replication solutions provided by Storage Vendors: If you are using a SAN, evaluate your
storage vendor's replication solutions.
The following table summarizes the RPO, RTO, and the pros and cons of each approach:
RPO
Dual Load
Periodic Incremental
Storage Replication
in progress
RTO
l
Built-in scripts
different configuration
l
database
compressed file transfers
Pros
l
Cons
Transparent to the
ETL licenses
system
More expensive
Media corruptions
are also replicated
to handle errors
Administrator's Guide
Recovering the Database
The failure of a node can invalidate the query plan for a currently running query. If the running query
does not require the failed node, the query continues to run. If a currently running query does require
the failed node, the query rolls back. HP Vertica then automatically reconfigures the query plan to
use alternate nodes and automatically retries the query. By default, HP Vertica makes three
attempts to rerun the query before canceling it.
Query performance within a cluster containing a failed node can deteriorate, as one node must now
do the work of two nodes. The extent of the performance impact varies depending on the queries
that you run and the location of the data that you are querying. Query performance remains affected
until the failed node has been completely restored.
Administrator's Guide
Monitoring HP Vertica
Monitoring HP Vertica
This section describes some of the ways in which you can monitor the health of your HP Vertica
database.
catalogpath
The catalog pathname specified when you created the database. See Creating a
databasename
nodename
The node name, as specified in the database definition. See Viewing a Database in
Administrator's Guide
Monitoring HP Vertica
See Also
l
Administrator's Guide
Monitoring HP Vertica
or
$ps -ef | grep -i vertica
$kill -USR1 process-id
Administrator's Guide
Monitoring HP Vertica
true
endscript
}
The following script is an example of the typical default setting for the dbLog file:
/mydb/dbLog {
# rotate weekly
weekly
# and keep for 52 weeks
rotate 52
# no complaining if vertica did not start yet
missingok
# compress log after rotation
compress
# this log is stdout, so rotate by copying it aside and truncating
copytruncate
}
For details about additional settings, issue the man logrotate command.
See Also
l
Administrator's Guide
Monitoring HP Vertica
You should see one HP Vertica process and one Spread process on each node for common
configurations. To monitor Administration Tools and connector processes:
$ps aux | grep vertica
There can be many connection processes but only one Administration Tools process.
Administrator's Guide
Monitoring HP Vertica
A high CPU percentage in top indicates that HP Vertica is CPU-bound. For example:
top - 11:44:28 up 53 days, 23:47, 9 users, load average: 0.91, 0.97, 0.81
Tasks: 123 total,
1 running, 122 sleeping,
0 stopped,
0 zombie
Cpu(s): 26.9%us, 1.3%sy, 0.0%ni, 71.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:
4053136 total, 3882020k used, 171116 free,
407688 buffers
Swap: 4192956 total,
176k used, 4192780 free, 1526436 cached
PID USER
PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
13703 dbadmin
1
0 1374m 678m 55m S 99.9 17.1
6:21.70 vertica
2606 root
16
0 32152 11m 2508 S 1.0 0.3
0:16.97 X
1 root
16
0 4748 552 456 S 0.0 0.0
0:01.51 init
2 root
RT -5
0
0
0 S 0.0 0.0
0:04.92 migration/0
3 root
34 19
0
0
0 S 0.0 0.0
0:11.75 ksoftirqd/0
...
The Tuple Mover runs automatically and thus consumes CPU time even if there are no
connections to the database.
The pdflush process (a set of worker threads for writing back dirty filesystem data) is
consuming a great deal of CPU time, possibly driving up the load. Adding RAM appears to
make the problem worse. Log in to root and change the Linux parameter swappiness to 0.
Administrator's Guide
Monitoring HP Vertica
Red Hat
3. Run the iostat utility. A high idle time in top at the same time as a high rate of blocks read in
iostat indicates that HP Vertica is disk-bound. For example:
$/usr/bin/iostat
Linux 2.6.18-164.el5 (qa01)
02/05/2011
avg-cpu: %user
%nice %system %iowait %steal
0.77
2.32
0.76
0.68
0.00
Device:
tps
Blk_read/s
Blk_wrtn/s
hda
0.37
3.40
10.37
sda
0.46
1.94
18.96
sdb
0.26
1.79
15.69
sdc
0.24
1.80
16.06
sdd
0.22
1.79
15.52
md0
8.37
7.31
66.23
%idle
95.47
Blk_read
2117723
1208130
1114792
1119304
1117472
4554834
Blk_wrtn
6464640
11816472
9781840
10010328
9676200
41284840
Description
DISK_STORAGE
Monitors the amount of disk storage used by the database on each node.
COLUMN_STORAGE
PROJECTION_STORAGE
Monitors the amount of disk storage used by each projection on each node.
Administrator's Guide
Monitoring HP Vertica
Note: An audit of columnar data includes any materialized columns in flex tables.
Administrator's Guide
Monitoring HP Vertica
The size of the database is reported in bytes. The AUDIT function also allows you to control the
accuracy of the estimated database size using additional parameters. See the entry for the AUDIT
function in the SQL Reference Manual for full details. HP Vertica does not count the AUDIT
function results as an official audit. It takes no license compliance actions based on the results.
Note: The results of the AUDIT function do not include flexible table data. Use the AUDIT_
FLEXfunction to monitor data usage for your Flex Tables license .
Targeted Auditing
If audits determine that the columnar table estimates are unexpectedly large, consider schemas,
tables, or partitions are using the most storage. You can use the AUDIT function to perform
targeted audits of schemas, tables, or partitions by supplying the name of the entity whose size you
want to find. For example, to find the size of the online_sales schema in the VMart example
database, run the following command:
VMart=> SELECT AUDIT('online_sales');
AUDIT
---------35716504
(1 row)
You can also change the granularity of an audit to report the size of each entity in a larger entity (for
example, each table in a schema) by using the granularity argument of the AUDIT function. See the
AUDIT function's entry in the SQL Reference Manual.
Administrator's Guide
Monitoring HP Vertica
The needle in the license meter adjusts to reflect the amount used in megabytes.
The Audit button returns the same information as the AUDIT() function in a graphical
representation.
The Details link within the License grid (next to the Audit button) provides historical information
about license usage. This page also shows a progress meter of percent used toward your
license limit.
Wed Nov
The check.txt is a file located in the /etc/keepalived/ directory, and it gets updated when you
submit changes to the kernel using sysctl -p, described in Disable the Address Resolution
Protocol (ARP). For example, the spread.pl script (see Configuring the Directors), writes to the
check.txt file, which is then modified to include only the remaining nodes in the event of a node
failure. Thus, the virtual server knows to stop sending vsql requests to the failed node.
You can also look for messages by issuing the following command at a shell prompt:
# tail -f /var/log/messages
Nov 3 09:21:00 p6 Keepalived: Starting Keepalived v1.1.17 (05/17,2010)Nov 3 09:21:00 p6
Keepalived: Starting Healthcheck child process, pid=32468
Nov 3 09:21:00 p6 Keepalived: Starting VRRP child process, pid=32469
Nov 3 09:21:00 p6 Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Nov 3 09:21:00 p6 Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Administrator's Guide
Monitoring HP Vertica
Administrator's Guide
Monitoring HP Vertica
Note: If multiple rebalance methods were used for a single table (for example, the table has
unsegmented and segmented projections), the table may appear multiple times - once for
each rebalance method.
REBALANCE_PROJECTION_STATUS can be used to gain more insight into the details for a
particular projection that is being rebalanced. It provides the same type of information as above,
but in terms of a projection instead of a table.
Monitoring Parameters
The following table describes the monitoring parameters for configuring HP Vertica.
Administrator's Guide
Monitoring HP Vertica
Parameters
Description
SnmpTrapDestinations
List
SnmpTrapsEnabled
SnmpTrapEvents
SyslogEnabled
SyslogEvents
Administrator's Guide
Monitoring HP Vertica
Parameters
Description
SyslogFacility
Administrator's Guide
Monitoring HP Vertica
Monitoring Events
To help you monitor your database system, HP Vertica traps and logs significant events that affect
database performance and functionality if you do not address their root causes. This section
describes where events are logged, the types of events that HP Vertica logs, how to respond to
these events, the information that HP Vertica provides for these events, and how to configure event
monitoring.
Description
vertica.log
All events are automatically posted to vertica.log. See Monitoring the Log
Files.
ACTIVE_EVENTS
This SQL system table provides information about all open events. See Using
System Tables and ACTIVE_EVENTS.
SNMP
Syslog
To log events to syslog, enable event reporting for each individual event you
want logged. See Configuring Event Reporting.
Administrator's Guide
Monitoring HP Vertica
Event
Name
Event
Type Description
Action
Low Disk
Space
possible.
Check dmesg to see what caused
the problem.
Also, use the DISK_RESOURCE_
REJECTIONS system table to
determine the types of disk space
requests that are being rejected and
the hosts on which they are being
rejected. See Managing Disk
Space within the Database
Administrators Guide for more
information about low disk space.
Read Only
File
System
access.
Administrator's Guide
Monitoring HP Vertica
Event
Name
Event
Type Description
Action
Current
Fault
Tolerance
at Critical
Level
Too Many
ROS
Containers
details.
Administrator's Guide
Monitoring HP Vertica
Event
Name
Event
Type Description
Action
WOS Over
Flow
Node State
Change
Recovery
Failure
specific situation.
Recovery
Error
specific situation.
Lock Error
10
Retrieval
Error
Administrator's Guide
Monitoring HP Vertica
Event
Name
Event
Type Description
Action
Refresh
11
Error
12
Lock Error
Tuple
13
Mover
Error
specific situation.
14
Service
scheduled task.
Task Error
Stale
15
Checkpoint
related to Moveout.
amount of data.
Event Data
To help you interpret and solve the issue that triggered an event, each event provides a variety of
data, depending upon the event logging mechanism used.
The following table describes the event data and indicates where it is used.
Administrator's Guide
Monitoring HP Vertica
vertica.log
ACTIVE_
EVENTS
(column
names)
SNMP
Syslog
Description
N/A
NODE_NAME
N/A
N/A
The node
where the
event
occurred.
Event Code
EVENT_CODE
A numeric ID
that indicates
the type of
event. See
Event Types
in the
previous
table for a list
of event type
codes.
Event Id
EVENT_ID
Event OID
Event Id
A unique
numeric ID
that identifies
the specific
event.
Administrator's Guide
Monitoring HP Vertica
vertica.log
ACTIVE_
EVENTS
(column
names)
SNMP
Syslog
Description
Event Severity
EVENT_
Event
Event Severity
The severity
SEVERITY
Severity
of the event
from highest
to lowest.
These events
are based on
standard
syslog
severity
types:
0
Emergency
1 Alert
2
Critical
3 Error
4 Warning
5 Notice
6 Info
7 Debug
PostedTimestamp
EVENT_
POSTED_
TIMESTAMP
N/A
PostedTimestamp
The year,
month, day,
and time the
event was
reported.
Time is
provided as
military time.
Administrator's Guide
Monitoring HP Vertica
vertica.log
ACTIVE_
EVENTS
(column
names)
SNMP
Syslog
Description
ExpirationTimestamp
EVENT_
N/A
ExpirationTimestamp
The time at
which this
EXPIRATION
event
expires. If the
same event
is posted
again prior to
its expiration
time, this
field gets
updated to a
new
expiration
time.
EventCodeDescription EVENT_
CODE_
description of
the event and
DESCRIPTION
details
pertinent to
the specific
situation.
ProblemDescription
EVENT_
Event
PROBLEM_
Short
description of
Description
the event.
DESCRIPTION
N/A
REPORTING_
NODE
Node
Name
ProblemDescription
N/A
A generic
The name of
the node
within the
cluster that
reported the
event.
Administrator's Guide
Monitoring HP Vertica
vertica.log
ACTIVE_
EVENTS
(column
names)
SNMP
Syslog
Description
DatabaseName
N/A
Database
DatabaseName
The name of
Name
the database
that is
impacted by
the event.
N/A
N/A
The name of
the host
within the
cluster that
reported the
event.
N/A
N/A
Event
Status
N/A
The status of
the event. It
can be either:
1 Open
2 Clear
Administrator's Guide
Monitoring HP Vertica
To disable event trapping for syslog, issue the following SQL command:
=> ALTERDATABASE mydb SET SyslogEnabled = 0;
Loss Of K Safety
Recovery Failure
Recovery Error
Administrator's Guide
Monitoring HP Vertica
Refresh Error
Stale Checkpoint
The following example generates a syslog entry for low disk space and recovery failure:
=> ALTER DATABASEmydb SET SyslogEvents = 'Low Disk Space, Recovery Failure';
auth
uucp (UUCP
subsystem)
cron
Administrator's Guide
Monitoring HP Vertica
daemon
subsystem)
l
See Also
l
Configuration Parameters
Administrator's Guide
Monitoring HP Vertica
used for the field names differs slightly. Under SNMP, the field names contain no spaces.
Also, field names are pre-pended with vert. For example, Event Severity becomes
vertEventSeverity.
When configuring your trap receiver, be sure to use the same hostname, port, and community
string you used to configure event trapping in HP Vertica.
Examples of network management providers:
n
IBM Tivoli
AdventNet
See Also
l
Configuration Parameters
Loss of K Safety
Administrator's Guide
Monitoring HP Vertica
Recovery Failure
Stale Checkpoint
CRC Mismatch
Note: After you complete steps 1 and 2 above, HP Vertica automatically traps the default
SNMP events. Only perform step 3 if you want to redefine which SNMP events are trapped.
HP recommends that you trap the Stale Checkpoint event even if you decide to reduce the
number events HP Vertica traps for SNMP. The specific settings you define have no effect on
traps sent to the log. All events are trapped to the log.
For example:
=> ALTER DATABASE mydb SET SnmpTrapDestinationList = 'localhost 162 public';
You can also specify multiple destinations by specifying a list of destinations, separated by
commas:
Administrator's Guide
Monitoring HP Vertica
Note: : Setting multiple destinations sends any SNMP trap notification to all destinations
listed.
Loss Of K Safety
Recovery Failure
Recovery Error
Refresh Error
Stale Checkpoint
CRC Mismatch
Administrator's Guide
Monitoring HP Vertica
SNMP
The following example illustrates a Too Many ROS Containers event posted to SNMP:
Administrator's Guide
Monitoring HP Vertica
Syslog
The following example illustrates a Too Many ROS Containers event posted and cleared within
syslog:
Aug 14 15:07:59 fc6-1 vertica: Event Posted: Event Code:4 Event Id:0 Event Severity:
Warning [4] PostedTimestamp: 2008-08-14 15:07:59.253729 ExpirationTimestamp:
2008-08-14 15:08:29.253729 EventCodeDescription: Too Many ROS Containers
ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: QATESTDB Hostname: fc61.verticacorp.com
Aug 14 15:08:54 fc6-1 vertica: Event Cleared: Event Code:4 Event Id:0 Event Severity:
Warning [4] PostedTimestamp: 2008-08-14 15:07:59.253729 ExpirationTimestamp:
2008-08-14 15:08:53.012669 EventCodeDescription: Too Many ROS Containers
ProblemDescription:
Too many ROS containers exist on this node. DatabaseName: QATESTDB Hostname: fc61.verticacorp.com
Administrator's Guide
Monitoring HP Vertica
Note: You cannot query system tables if the database cluster is in a recovering state. The
database refuses connection requests and cannot be monitored. HP Vertica also does not
support DDL and DML operations on system tables.
These schemas reside in the default search path so there is no need to specify schema.table in
your queries unless you change the search path to exclude V_MONITOR or V_CATALOG or both.
The system tables that make up the monitoring API are described fully in the SQL Reference
Manual. You can also use the following command to view all the system tables and their schema:
Administrator's Guide
Monitoring HP Vertica
System information
System resources
Background processes
HP Vertica reserves some memory to help monitor busy systems. Using simple system table
queries makes it easier to troubleshoot issues. See also SYSQUERY and SYSDATA pools under Built-in
pools topic in SQL Reference Manual.
Note: You can use external monitoring tools or scripts to query the system tables and act upon
the information, as necessary. For example, when a host failure causes the K-safety level to
fall below the desired level, the tool or script can notify the database administrator and/or
appropriate IT personnel of the change, typically in the form of an e-mail.
CREATE
CREATE
CREATE
INSERT
SCHEMA SS;
TABLE SS.TT (c1 int);
PROJECTION SS.TTP1 AS SELECT * FROM ss.tt UNSEGMENTED ALL NODES;
INTO ss.tt VALUES (1);
Administrator's Guide
Monitoring HP Vertica
Examples
The following query uses the VMart example database (see Introducing the VMart Example
Database) to obtain the number of rows and size occupied by each table in the database.
=> SELECT t.table_name AS table_name,
SUM(ps.wos_row_count + ps.ros_row_count) AS row_count,
SUM(ps.wos_used_bytes + ps.ros_used_bytes) AS byte_count
FROM tables t
JOIN projections p ON t.table_id = p.anchor_table_id
JOIN projection_storage ps on p.projection_name = ps.projection_name
WHERE (ps.wos_used_bytes + ps.ros_used_bytes) > 500000
GROUP BY t.table_name
ORDER BY byte_count DESC;
table_name
| row_count | byte_count
--------------------+-----------+-----------online_sales_fact |
5000000 | 171987438
store_sales_fact
|
5000000 | 108844666
store_orders_fact |
300000 |
9240800
product_dimension |
60000 |
2327964
customer_dimension |
50000 |
2165897
inventory_fact
|
300000 |
2045900
(6 rows)
The rest of the examples illustrate simple ways to use system tables in queries.
=> SELECT table_name FROM columns WHERE data_type ILIKE 'Numeric' GROUP BY table_name;
table_name
-----------n1
(1 row)
=> SELECT current_epoch, designed_fault_tolerance, current_fault_tolerance
FROM SYSTEM;
current_epoch | designed_fault_tolerance | current_fault_tolerance
---------------+--------------------------+------------------------492 |
1 |
1
(1 row)
=> SELECT node_name, total_user_session_count, executed_query_count FROM
Administrator's Guide
Monitoring HP Vertica
query_metrics;
node_name | total_user_session_count | executed_query_count
-----------+--------------------------+---------------------node01
|
53 |
42
node02
|
53 |
0
node03
|
42 |
120
node04
|
53 |
0
(4 rows)
=> SELECT table_schema FROM primary_keys;
table_schema
-------------public
public
public
public
public
public
public
public
public
store
online_sales
online_sales
(12 rows)
Administrator's Guide
Monitoring HP Vertica
Data Collector
HP Vertica also provides a utility called the Data Collector (DC), which collects and retains history
of important system activities and records essential performance and resource utilization counters.
Data Collector extends system table functionality by:
l
You can use the information the Data Collector retains to query the past state of system tables and
extract aggregate information, as well as do the following:
l
DC does not collect data for nodes that are down, so no historical data would be available for that
node.
Data Collector works in conjunction with the Workload Analyzer (WLA), an advisor tool that
intelligently monitors the performance of SQL queries and workloads and recommends tuning
actions based on observations of the actual workload history. See Analyzing Workloads for more
information about WLA.
Administrator's Guide
Monitoring HP Vertica
DC Tables
Caution: Data Collector tables (prefixed by dc_) reside in the V_INTERNAL schema and are
provided for informational purposes only. They are provided as-is and are subject to removal or
change without notice. If you use Data Collector tables in scripts or monitoring tools, you might
need to change your scripts and tools after an HP Vertica upgrade. HP recommends that you
use the Workload Analyzer instead of accessing the Data Collector tables directly.
See Also
l
DATA_COLLECTOR
ANALYZE_WORKLOAD
TUNING_RECOMMENDATIONS
Administrator's Guide
Monitoring HP Vertica
See Also
l
The following command returns the retention policy for a specific component, NodeState.
=> SELECT get_data_collector_policy('NodeState');
The results let you know that 10KB is retained in memory and 100KB on disk:
get_data_collector_policy------------------------------------------
Administrator's Guide
Monitoring HP Vertica
A superuser can modify policies, such as change the amount of data to retain, by invoking the
SET_DATA_COLLECTOR_POLICY() function, as follows:
SET_DATA_COLLECTOR_POLICY('component', 'memoryKB', 'diskKB' )
The SET_DATA_COLLECTOR_POLICY() function sets the retention policy for the specified
component on all nodes, and lets you specify memory and disk size to retain in kilobytes. Failed
nodes receive the new setting when they rejoin the cluster.
For example, the following statement specifies that the NodeState component be allocated 50KB
of memory and 250KB of disk space:
=> SELECT SET_DATA_COLLECTOR_POLICY('NodeState', '50', '250');
SET_DATA_COLLECTOR_POLICY
--------------------------SET
(1 row)
Before you change a retention policy, you can view the current setting by calling the GET_DATA_
COLLECTOR_POLICY() function.
You can also use the GET_DATA_COLLECTOR_POLICY() function to verify changed settings.
For example, the following query retrieves a brief statement about the retention policy for the
NodeState component:
=> SELECT GET_DATA_COLLECTOR_POLICY('NodeState');
GET_DATA_COLLECTOR_POLICY
-----------------------------------------50KB kept in memory, 250KB kept on disk.
(1 row)
Tip: If you do not know the name of a component, you can query the V_MONITOR.DATA_
Administrator's Guide
Monitoring HP Vertica
COLLECTOR system table to get a a full list. For example, the following query returns all
current Data Collector components and a description of each: => SELECT DISTINCT
component, description FROM data_collector ORDER BY 1 ASC;
See Also
l
GET_DATA_COLLECTOR_POLICY
SET_DATA_COLLECTOR_POLICY
CREATE_<component>_TABLE.sql contains the SQL DDL needed to create a table into which
Data Collector logs for the component can be loaded.
CLEAR_DATA_COLLECTOR()
FLUSH_DATA_COLLECTOR()
V_MONITOR.DATA_
COLLECTOR
Administrator's Guide
Monitoring HP Vertica
The following command clears the Data Collector records for all components on all nodes:
=> SELECT clear_data_collector();
clear_data_collector
---------------------CLEAR
(1 row)
Note: After you clear the DataCollector log, the information is no longer available for querying.
To flush records on a single component, pass a component argument to the function. For example,
the following command flushes the ResourceAcquisitions component:
Administrator's Guide
Monitoring HP Vertica
See Also
l
DATA_COLLECTOR
Administrator's Guide
Monitoring HP Vertica
Related Topics
V_MONITOR.DATA_COLLECTOR and Data Collector Functions in the SQL Reference Manual
Retaining Monitoring Information and How HP Vertica Calculates Database Size in this guide
Administrator's Guide
Monitoring HP Vertica
build_tag
Note: Only the rows where the PROJECTION_REFRESHES.IS_EXECUTING column equals false
are cleared.
Administrator's Guide
Monitoring HP Vertica
See Also
l
CLEAR_PROJECTION_REFRESHES
PROJECTION_REFRESHES
V_MONITOR.PARTITIONS displays NULLS in the partition_key column for any ROS's that
have not been reorganized.
Administrator's Guide
Monitoring HP Vertica
Examples
This example shows how to find the various resource pools defined in the system.
=> SELECT name, memorysize, maxmemorysize FROM V_CATALOG.RESOURCE_POOLS;
name
| memorysize | maxmemorysize
----------+------------+--------------general |
| Special: 95%
sysquery | 64M
|
sysdata | 100M
| 10%
wosdata | 0%
| 25%
tm
| 200M
|
refresh | 0%
|
recovery | 0%
|
dbd
| 0%
|
jvm
| 0%
| 10%
(9 rows)
Administrator's Guide
Monitoring HP Vertica
ceo_pool
ad hoc_pool
billing_pool
web_pool
batch_pool
dept1_pool
dept2_pool
(8 rows)
|
|
|
|
|
|
|
250M
200M
0%
25M
150M
0%
0%
|
| 200M
|
|
| 150M
|
|
|
|
|
|
|
|
|
10
0
0
10
0
5
8
|
|
|
|
|
|
|
3
5
10
Administrator's Guide
Monitoring HP Vertica
memory_size_actual_kb
| 204800
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 6 ]--------------+--------pool_name
| refresh
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 7 ]--------------+--------pool_name
| recovery
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 8 ]--------------+--------pool_name
| dbd
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
-[RECORD 9 ]--------------+--------pool_name
| jvm
memory_size_kb
| 0
memory_size_actual_kb
| 0
memory_inuse_kb
| 0
general_memory_borrowed_kb | 0
running_query_count
| 0
Administrator's Guide
Monitoring HP Vertica
pool_name
| general
thread_count
| 12
open_file_handle_count | 18
memory_inuse_kb
| 708504
queue_entry_timestamp | 2013-12-04 12:55:38.566614-05
acquisition_timestamp | 2013-12-04 12:55:38.566623-05
-[RECORD 9 ]----------+-----------------------------...
You can determine how long a query waits in the queue before it can run. To do so, you obtain the
difference between the acquisition_timestamp and the queue_entry_timestamp using a query
as this example shows:
=> SELECT pool_name, queue_entry_timestamp, acquisition_timestamp,
(acquisition_timestamp-queue_entry_timestamp) AS 'queue wait'
FROM V_MONITOR.RESOURCE_ACQUISITIONS WHERE node_name ILIKE '%node0001';
-[RECORD 1 ]---------+-----------------------------pool_name
| sysquery
queue_entry_timestamp | 2013-12-05 07:07:08.815362-05
acquisition_timestamp | 2013-12-05 07:07:08.815367-05
queue wait
| 00:00:00.000005
-[RECORD 2 ]---------+-----------------------------pool_name
| sysquery
queue_entry_timestamp | 2013-12-05 07:07:14.714412-05
acquisition_timestamp | 2013-12-05 07:07:14.714417-05
queue wait
| 00:00:00.000005
-[RECORD 3 ]---------+-----------------------------pool_name
| sysquery
queue_entry_timestamp | 2013-12-05 07:09:57.238521-05
acquisition_timestamp | 2013-12-05 07:09:57.281708-05
queue wait
| 00:00:00.043187
-[RECORD 4 ]---------+-----------------------------...
See Also
l
See the SQL Reference Manual for detailed descriptions of the monitoring tables.
See Monitoring and Configuring Resource Pools in Management Console for descriptions of
other ways to monitor resource usage.
Monitoring Recovery
There are several ways to monitor database recovery:
Administrator's Guide
Monitoring HP Vertica
System tables
Administrator's Guide
Monitoring HP Vertica
RECOVERY_STATUS
PROJECTION_RECOVERIES
Specifically, the recovery_status system table includes information about the node that is
recovering, the epoch being recovered, the current recovery phase, and running status:
=>select node_name, recover_epoch, recovery_phase, current_completed, is_running from
recovery_status;
node_name | recover_epoch | recovery_phase
| current_completed | is_running
-----------+---------------+-------------------+-------------------+-------------node01
|
|
| 0
| f
node02
| 0
| historical pass 1 | 0
| t
node03
| 1
| current
| 0
| f
To see a single record from the projection_recoveries system table, add limit 1 to the query.
Note: You can also monitor the state of your database nodes on the Management Console
Overview page under the Database section, which tells you the number of nodes that are up,
critical, recovering, or down. To get node-specific information, click Manage at the bottom of
the page.
Administrator's Guide
Monitoring HP Vertica
See Also
l
Monitoring HP Vertica
Administrator's Guide
Monitoring HP Vertica
Multiple database cluster states and key performance indicators that report on the cluster's
overall health
Query concurrency and internal/user sessions that report on important events in time
Cluster-wide messages
Administrator's Guide
Monitoring HP Vertica
Tip: You can increase the value for MaxClientSessions on an MC-monitored database to take
extra sessions into account. See Managing Sessions for details.
Tasks
Operations you can perform on MC are grouped into the following task-based areas:
l
Databases and Clusters. Create, import, manage and monitor one or more databases on one
or more clustersthis includes creating new empty databases and importing existing database
clusters into the MC interface. See Managing Database Clusters on MC.
MC Settings. Configure MC and user settings, as well as use the MC interface to install HP
Vertica on a cluster of hosts. See Managing MC Settings.
Administrator's Guide
Monitoring HP Vertica
Message Center. View, sort, and search database messages and optionally export messages
to a file. See Monitoring MC-managed Database Messages.
Diagnostics. View and resolve MC-related issues, as well as browse HP Vertica agent and
audit logs. See Troubleshooting Management Console.
Recent Databases
The area directly below Tasks displays all databases that you created on or imported into the MC
interface. You can install one or more databases, on the same cluster or on different clusters, but
you can have only one database running on a single cluster at a time. UP databases appear in green
and DOWN databases are red. An empty space under Recent Databases means that you have not
yet created or imported a database into the MC interface.
Administrator's Guide
Monitoring HP Vertica
Quick Stats
The Quick Stats sidebar on the right of the page provides instant alerts and information about your
cluster's status.
Administrator's Guide
Monitoring HP Vertica
Messages display the number of unread messages and alerts for the database. This count
differs from the number of total messages across all your databases. Click the value to open the
Message Center.
Low Disk Space alerts you to the number of nodes that are low on disk space. Click the value to
go to the Manage page. On the Manage page, the Storage Used KPIView is displayed.
Workload Analyzer (WLA) analyzes system information retained in SQL system tables and
provides tuning recommendations through a SQL command you can run, along with the cost
(low, medium, or high) of running the command. See Analyzing Workloads for more information.
Administrator's Guide
Monitoring HP Vertica
Database Nodes Health displays the state of your nodes. Click a node value to open the
Manage page.
I/O Wait Notices displays the number of nodes that, in the last hour, have recorded Disk I/O
waits and Network I/O waits exceeding the wait threshold (1 second for Disk and 0 seconds for
Network).
Query processing displays the number of running and queued queries. Click a query value to
open the Query Monitoring page.
Projections displays the number of projections, the number of unsegmented projections, and the
number of unsafe projections for the schema in the database with the most projections. Click a
projection value to open the Table Treemap chart in the Activity page.
License Consumption displays the number of licenses your database uses, and the percentage
of your HP Vertica Community Edition or Enterprise Edition license being used.
Status Summary
The Status Summary tab displays four charts that provide a general overview of the status of your
cluster.
The CPU/Memory Usage module is a graph-based overview of cluster resources during the last
hour. The chart displays the number of nodes in the database cluster and plots average and pernode percentages for CPU, memory, and disk I/O usage. Click a data point (that represents a
Administrator's Guide
Monitoring HP Vertica
node), and the Node Details page opens. See Monitoring Cluster CPU/Memory for more
information.
The 'General Pool' Free Memory module displays GENERAL pool activity in the last 24 hours.
The chart displays average query queue times, average GENERAL pool free memory, and resource
rejections. Click a data point, and the Resource Pools Monitoring chart opens. Use this chart to see
how much free memory there is in GENERALpool, or if there have been unusually high queue
times in the past day. See Managing Workloads for information.
The Database Nodes Health module displays the states of the nodes in your cluster. Nodes in the
chart are displayed as down, critical, recovering, or up. However, note that critical and recovering
nodes are still considered "up" by the database. Nodes in a state of recovery indicate a recovery
operation is in progress. See Failure Recovery for more information. Click a node count number
above the chart to open the Manage page.
The Query Processing Activity module displays a count of completed and failed queries in the
last 24 hours. The bar chart displays running and queued queries. The statistics above the chart
show the running and queued query values displayed in the chart, as well as total completed and
total failed queries in the last 24 hours. Click a query count number above the chart to open the
Monitoring Running Queries chart.
System Health
The System Health tab provides a comprehensive summary of your system resource usage and
node information, with configurable statistics that allow you to specify acceptable ranges of
resource usage.
Administrator's Guide
Monitoring HP Vertica
The Memory Usage chart displays the number of nodes falling outside of the specified range of
acceptable memory usage. Move the sliders to configure the minimum and maximum percentage of
memory usage per node. Hover your cursor over the Low and High values to see lists of what nodes
fall, respectively, below or above the acceptable range. Click a node value to go to the Manage
page. On the Manage page, the Memory Utilization KPIView is displayed.
The Spread Retransmission Rate chart displays the number of nodes that exceed the specified
spread retransmission rate threshold. When a node's retransmission rate is too high, it is not
communicating properly with other nodes. Move the slider to configure the maximum acceptable
retransmission rate. Hover your cursor over the Nodes value to see a list of what nodes exceeded
the acceptable threshold. Click the node value to view spread retransmit rate alerts in the Message
Center.
The CPUUsage chart displays the number of nodes that exceed the specified range of acceptable
processor usage. Move the sliders to configure the minimum and maximum percentage of CPU
usage per node. Hover your cursor over the Low and High values to see lists of what nodes are
below or above the specified range. Click a node value to go to the Manage page. On the Manage
page, the CPU Utilization KPIView is displayed.
The Reboot Rate chart displays the number of times nodes in the cluster have rebooted within the
specified amount of time. You can use this to see if node have gone down recently, or if there have
been an unusual number of reboots within a period of time. Move the slider to configure the length of
time in days. Hover over the Times value to see a list of the nodes that have rebooted and the times
at which they did so.
The Disk Space Usage chart displays the number of nodes falling outside of the range of
acceptable disk space usage. Move the slider to specify the displayed range of acceptable memory
usage per node. Hover your cursor over the Nodes value to see a list of what nodes exceed the
acceptable range. Click the nodes value to go to the Manage page. On the Manage page, the
Storage Used KPIView is displayed.
The Cluster Clock Skew Rate module displays the number of nodes that exceed a clock skew
threshold. Nodes in a cluster whose clocks are not in sync can interfere with time-related database
functions, the accuracy of database queries, and Management Console's monitoring of cluster
activity.
Query Synopsis
The Query Synopsis page provides two charts that report system query activity and resource pool
usage.
Administrator's Guide
Monitoring HP Vertica
The Query Statistics module displays four charts that provide an overview of running queries and
queued queries, as well as completed queries and failed queries in the past 24 hours. The charts
display the number of queries in each category by resource pool. Hover over a section of the chart
to view the number of queries in that resource pool. Click a section of the chart (Running, Queued,
Completed, or Failed) to open the corresponding time in the Query Monitoring activity chart.
The Query Distribution chart provides an overview of user and system query activity and reports
the type of operation that ran, which MC gathers from the production database. Hover your cursor
over chart points for more details. To zoom, adjust the slider bar at the bottom of the chart around an
area of interest. Double click any point in the graph to open the Queries chart.
Administrator's Guide
Monitoring HP Vertica
Refresh the view from information MC gathers from the production database
View key performance indicators (KPI) on node state, CPU, memory, and storage utilization
(see Monitoring Cluster Performance for details)
Note: Starting, stopping, adding, and dropping nodes and rebalancing data across nodes
works with the same functionality and restrictions as those same tasks performed through the
Administration Tools.
Administrator's Guide
Monitoring HP Vertica
Database Cluster State, you see that all nodes are UP), click the Refresh button in the toolbar.
Doing so forces MC to immediately synchronize with the agents and update MC with new data.
Don't press the F5 key, which redisplays the page using data from MC and ignores data from the
agent. It can take several seconds for MC to enable all database action buttons.
See Also
l
See Also
l
Administrator's Guide
Monitoring HP Vertica
Metrics on the database are computed and averaged over the latest 30 seconds of activity and
dynamically updated on the cluster grid.
Node state(default view) shows node status (up, down, k-safety critical) by color; you can
filter which nodes appear on the page by sliding the health filter from left to right
After you make a selection, there is a brief delay while MC transmits information back to the
requesting client. You can also click Sync in the toolbar to force synchronization between MC and
the client.
Administrator's Guide
Monitoring HP Vertica
If the unhealthy node were to consume additional resources, its color would change from a dull
orange-red to a brighter red.
Filtering is useful if you have many nodes and want to see only the ones that need attention, so you
can quickly resolve issues on them.
Queries
Internal Sessions
User Sessions
Memory Usage
System Bottlenecks
Table Treemap
Query Monitoring
Administrator's Guide
Monitoring HP Vertica
Hover over a data point for more information about percent usage for each of the resource types.
Administrator's Guide
Monitoring HP Vertica
If you click a data point, MC opens a details page for that point in time, summarizing the number of
user queries and system queries. This page can help you identify long-running queries, along with
the query type. You can sort table columns and export the report to a file.
Administrator's Guide
Monitoring HP Vertica
To return to the main Queries page, click <database name> Activity in the navigation bar.
Administrator's Guide
Monitoring HP Vertica
CategoryFilter which internal session types (moveout, mergeout, rebalance cluster) appear in
the graph. The number in parentheses indicates how many sessions are running on that
operation.
Session durationLists time, in milliseconds, for all sessions that appear on the graph. The
minimum/maximum values on which you can filter (0 ms to n ms) represent the
minimum/maximum elapsed times within all sessions currently displayed on the graph. After
you choose a value, the chart refreshes to show only the internal sessions that were greater than
or equal to the value you select.
Records requestedRepresents the total combined sessions for the Category and Session
Duration filters.
Yellow bars represent user (system) sessions. If you click a yellow bar, MC opens a Detail page
that shows all queries that ran or are still running within that session.
Blue bars represent user requests (transactions within the session). If you click a blue section in
Administrator's Guide
Monitoring HP Vertica
the graph, MC opens a Detail page that includes information for that query request only.
When you hover your mouse over a transaction bar, a dialog box provides summary information
about that request, such as which user ran the query, how long the transaction took to complete,
or whether the transaction is still running.
Chart Results
Extremely busy systems will show a lot of activity on the interface, perhaps more than you can
interpret at a glance. You can filter session results by selecting a smaller number of records, a
specific user, or by changing the session duration (how long a session took to run).
As you apply filters, the Records requested number changes to represent the total combined
sessions based on your filters:
l
Session durationlist time, in milliseconds, for all the sessions that appear on the graph. The
minimum/maximum values on which you can filter (0 ms to n ms) represent the
minimum/maximum elapsed times within all sessions currently displayed on the graph. After
you choose a value, the chart refreshes to show only the sessions that were greater than or
equal to the value you select.
Records requestedrepresents the total combined user sessions for the User and Session
duration filters.
Administrator's Guide
Monitoring HP Vertica
move the list out of the way by dragging it to another area on the page.
swap
free
buffer
other (memory in use by all other processes running on the system besides the main Vertica
process, such as the MC process or agents)
HP Vertica
catalog
When you hover over a data point, a dialog box displays percentage of memory used during that
time period for the selected node.
Administrator's Guide
Monitoring HP Vertica
Example
The following image shows that, over the last three hours, the most-used resources on the selected
node were disk I/O, CPU, and memory. Line colors map to the components in the legend.
According to the above chart, just before 07:40 the maximum resources used changed from CPU to
memory, denoted by a change from green to pink and a new data point. The following image
provides a closeup of the change.
Administrator's Guide
Monitoring HP Vertica
Each bar, bound by a gray box, represents an individual query. Within a query, a different color
represents each query phase. The chart does not show phases for queries with durations of less
than 4 seconds. Blank spaces within a query represent waiting times, as in the image below.
Hover over a phase in the query for information on the phase type and duration.
The chart shows queries run over the last 15 minutes. The chart automatically refreshes every five
minutes, unless you clear the Auto Refresh option in the toolbar.
Administrator's Guide
Monitoring HP Vertica
Administrator's Guide
Monitoring HP Vertica
Depending on the number of tables in your database, the chart may be crowded. Use the filters to
the right of the chart to show only the 100 largest or smallest tables, or filter the chart by schema.
Administrator's Guide
Monitoring HP Vertica
The projections summary displays information about projections by schema. Click the projections
number to see a list of the specified projections.
l
Total projections.
Segmented projections, the number of projections that are segmented across multiple nodes.
Unsegmented projections, the number of projections that are not segmented across multiple
nodes.
Projections with Distribution Skew, the number of projections unevenly distributed across
nodes. Tables with fewer than 1000 rows are not counted. Move the slider to configure the
acceptable distribution skew percentage.
The number of projections with more than a specified number of containers per node. Move the
slider to change the minimum number of containers.
Unsafe projections, the number of projections with a K-safety less than the database's K-safety.
Unused projections.
Not up to date projections, which are ineligible to participate in query execution. Projections on
tables with loaded date are marked not up-to-date until refreshed.
Administrator's Guide
Monitoring HP Vertica
Hover over a table on the chart to get individual table information, which appears on the right.
Click the table or type a table name in the text box to the right of the chart to go to the Table Details
page. The Table Details page displays information on the table's properties and columns, and about
projections for the table. To the right, bar charts provide information, by node, on the amount of
storage, number of deleted rows, and number of deleted vectors.
Administrator's Guide
Monitoring HP Vertica
Administrator's Guide
Monitoring HP Vertica
The Query Monitoring page includes four tables, displayed as tabs, that list running queries, queued
queries, successfully run queries, and failed queries. Recent query statistics appear above the
tabs. Use the time frame list above the tabs to change the time frame for the statistics, and to filter
the displayed set of failed and completed queries displayed in the tabs. From the Action column,
click Cancel to cancel a running or queued query. Click Profile to launch a new window that
displays the Explain Plan page.
Charts at the bottom of the page display aggregate query usage by node or user. Hover over a bar
with your cursor to see its value. When sorted by value, the left-most bar in each chart represents
the node or user with the highest value.
The Query Monitoring page refreshes every 20 seconds by default. To change the refresh interval,
click the Page Settings button in the upper-right corner of the page. A dialog appears. Type the new
refresh interval, in milliseconds, in the text box.
Administrator's Guide
Monitoring HP Vertica
In the Failed Queries chart, click the plus symbol next to a failed query to see the failure details for
each node involved in the query's execution.
To profile the query, click Profile in the Actions column. The Explain Plan page launches in a
second window and profiles the query.
To export the data in one of the Query Monitoring tables, click the table's tab, then click the Export
button in the upper-right corner of the page. The browser downloads the data for that chart in a .dat
file. The data exported will include columns that may not be visible in the MC, including the
minimum and maximum values for Memory, Thread Count, and Open File Handles.
Administrator's Guide
Monitoring HP Vertica
For a more detailed view of node activity, use the mouse to drag-select around a problem area in
one of the graphs, such as the large spike in network traffic in the above image. Then hover over the
high data point for a summary.
Administrator's Guide
Monitoring HP Vertica
See Also
l
Administrator's Guide
Monitoring HP Vertica
If you click the Details link MC opens the Message Center, which lists up to 500 of the most
current messages and reports on the following database-related conditions:
l
Loss of K-safety
WOS overflow
Recovery error
Recovery failure
Administrator's Guide
Monitoring HP Vertica
Refresh error
Message Severity
Messages are color coded to represent the level of severity. For example:
l
Orange - Error
Yellow - Warning
Green - Notice
Blue - Informational
The icons that correspond with a message on the interface use the same color conventions.
Administrator's Guide
Monitoring HP Vertica
After you click OK, the Message Center returns up to 500 messages and updates the toolbar with
your search criteria. By default, the Message Center displays, "showing recent entries, all alert
types," but if you requested specific message types the toolbar displays "alert types filtered." The
toolbar will also show the date range if you supplied one.
Administrator's Guide
Monitoring HP Vertica
Based on the likelihood that MC will find more than 500 messages, a date search from <date> to
<now> will return a different set of messages than it would from a search recent entries request.
After you specify a date range, click Done to close the calendar, and then click OK to see the
results in the Message Center.
Administrator's Guide
Monitoring HP Vertica
Client-side filtering works only on messages that MC has already returned, whether you can see
them or not. If you select or clear message types in this filter, the toolbar status does not change.
Client-side selections also do not change based on the choices you make in your server-side
search. The goal of client-side filtering is to let you further refine server-side results.
Administrator's Guide
Monitoring HP Vertica
Where you
Message can export on
type
MC
All dbrelated
Message
Center page
System-generated filename
Contents of
exported file
vertica-alerts-<timestamp>.csv
Exports messages in
the Message Center
message
to a .csv file.
types
Message contents
are saved under the
following headings:
l
Create time
Severity
Database
Summary (of
message)
Description (more
details)
MC log
files
Diagnostics
page
mconsole-<timestamp>.log
Exports MC log
search results from
MC to a .log file
under the following
headings:
l
Time
Type (message
severity)
Component (such
as TM, Txn,
Recover, and so
on)
Message
Administrator's Guide
Monitoring HP Vertica
Where you
Message can export on
type
MC
System-generated filename
Contents of
exported file
Vertica
vertica-vertica-<db>-<timestamp>.log
logs
Manage page
Double-click
MC to a .log file
to the details
headings:
Time
Type (message
tab.
severity)
l
Component (such
as TM, Txn,
Recover, and so
on)
Agent
logs
Manage page
vertica-agent-<db>-<timestamp>.log
Message
Time
Type (message
Log tab.
severity)
l
Component (such
as TM, Txn,
Recover, and so
on)
Message
Administrator's Guide
Monitoring HP Vertica
Where you
Message can export on
type
MC
System-generated filename
Contents of
exported file
Query
vertica-querydetails-<db>-<timestamp>.dat
details
Activity page
Click any
between
query spike in
<timestamp> and
<timestamp> as a
to the Detail
page.
Content is saved
under the following
headings:
MC user
activity
Diagnostics
page
vertica_audit<timestamp>.csv
Query type
Session ID
Node name
Started
Elapsed
User name
Request/Query
Log task
Time
MC User
Resource
Target User
Client IP
Activity
Administrator's Guide
Monitoring HP Vertica
Database creation
Start/stop a database
Drop a database
Database settings changes (individual settings are tracked in the audit record)
Syncing the database with the cluster (who clicked Sync on grid view)
Management Console setting changes (individual settings are tracked in the audit record)
Administrator's Guide
Monitoring HP Vertica
Administrator's Guide
Analyzing Workloads
Analyzing Workloads
If your queries are performing suboptimally, you can get tuning recommendations for them, as well
as for hints about optimizing database objects, by using the Workload Analyzer (WLA).
See Understanding WLA Triggering Conditions for the most common triggering conditions and
recommendations.
table_name
A specific table
schema_name
Administrator's Guide
Analyzing Workloads
The optional since_time argument specifies the WLA to return values from all scope events
starting at since_time and continuing to the current system status. If you omit since_time,
ANALYZE_WORKLOAD returns recommendations for events since the last recorded time that you
called the function. You must explicitly cast the since_time string value to either TIMESTAMP or
TIMESTAMPTZ.
The following examples show four ways to express the since_time argument with different
formats. All queries return the same result for workloads on table t1 since October 4, 2012:
=>
=>
=>
=>
SELECT
SELECT
SELECT
SELECT
ANALYZE_WORKLOAD('t1',
ANALYZE_WORKLOAD('t1',
ANALYZE_WORKLOAD('t1',
ANALYZE_WORKLOAD('t1',
The next invocation of ANALYZE_WORKLOAD() analyzes events from this point forward.
Administrator's Guide
Analyzing Workloads
Administrator's Guide
Analyzing Workloads
the Database Designer on two tables and consider setting a user's password in record 3. Note that
RECORD 3 also provides the ALTER USER command to run because the tuning action is a SQL
command.
A LOW cost has minimal impact on resources from running the tuning command. You can
perform the tuning operation at any time, like changing the user's password in Record 3 above.
A MEDIUM cost has moderate impact on resources from running the tuning command
A HIGH cost has maximum impact on resources from running the tuning command. Depending
on the size of your database or table, consider running high-cost operations after hours instead
of during peak load times.
Examples
The following statement tells WLA to analyze all events for a table named locations:
=> SELECT ANALYZE_WORKLOAD('locations');
The recommendation s that you run the Database Designer on the table, an operation that,
depending on the size of the locations table, could potentially have a high cost, so you might want
to run Database Designer after hours.
-[RECORD 1 ]----------+-----------------------------------------------observation_count
| 1
first_observation_time |
last_observation_time |
tuning_parameter
| public.locations
tuning_description
| run database designer on table public.locations
tuning_command
|
tuning_cost
| HIGH
The following statement analyzes workloads on all tables in the VMart example datbase (see
Introducing the VMart Example Database) since one week before today:
=> SELECT ANALYZE_WORKLOAD('', NOW() - INTERVAL '1 week');
Workload Analyzer found two issues which are described below the sample output.
Administrator's Guide
Analyzing Workloads
-[RECORD 1 ]----------+-----------------------------------------------------observation_count
| 4
first_observation_time | 2012-02-17 13:57:17.799003-04
last_observation_time | 2011-04-22 12:05:26.856456-04
tuning_parameter
| store.store_orders_fact.date_ordered
tuning_description
| analyze statistics on table column store.store_orders_fact.date_
ordered
tuning_command
| select analyze_statistics('store.store_orders_fact.date_
ordered');
tuning_cost
| MEDIUM
-[RECORD 2 ]---------+-----------------------------------------------------...
-[RECORD 14 ]---------+----------------------------------------------------observation_count
| 2
first_observation_time | 2012-02-19 17:52:03.022644-04
last_observation_time | 2012-02-19 17:52:03.02301-04
tuning_parameter
| SELECT x FROM t WHERE x > (SELECT SUM(DISTINCT x) FROM
| t GROUP BY y) OR x < 9;
tuning_description
| consider incremental design on query
tuning_command
|
tuning_cost
| HIGH
In RECORD 14, WLA identifies an underperforming query and recommends that you use the
Database Designer to run an incremental design. This information is shown in the tuning_
parameter column. Note that the potential cost of running this operation is HIGH.
This statement analyzes workloads on all tables two days before today (now):
=> SELECT ANALYZE_WORKLOAD('', NOW() - '2 DAYS'::INTERVAL);
System information that WLA uses for its recommendations is held in SQL system tables, so
querying the TUNING_RECOMMENDATIONS system table does not run the Workload Analyzer.
Administrator's Guide
Analyzing Workloads
See Also
l
Administrator's Guide
Analyzing Workloads
Management Console displays five recommendations per page and includes a message at bottom
left, which lets you know where you are within the total number of recommendations; for example,
"Showing n-nn of nn entries." Depending on the total number of recommendations, you can move
forward and backward through them by clicking the arrows at bottom right.
Tip: You can force the WLA task to run immediately by clicking Analyze over the Cost
column.
For additional information about tuning recommendations and their triggering event, see
Understanding WLA Triggering Conditions.
See Also
l
ANALYZE_WORKLOAD
Administrator's Guide
Analyzing Workloads
In rare circumstances, tuning recommendation WLA proposes might not resolve the underlying
problem. Because you might occasionally need help understanding WLA's recommendations, the
following table lists some of the most common triggering conditions, along with the
recommendations to resolve the issue and a pointer to more information, when available.
The
triggering
condition
What to do about it
Where to get
more
information
Internal
See ALTER
configuratio
DATABASE
n parameter
is not the
same
across
nodes.
Administrator's Guide
Analyzing Workloads
The
triggering
condition
What to do about it
Where to get
more
information
An unused
See the
projection
following
meets the
topics in the
following
SQL
conditions:
Reference
Manual:
No
queries
DROP
on
PROJECT
projectio
ION
n for
more
PROJECT
ION_
than 30
STORAGE
days but,
projectio
l
n's
PROJECT
IONS
anchor
table has
TABLES
been
queried
more
than 10
times
l
Projectio
n's
anchor
table is
not a
temp or
system
table
Projectio
n's table
is not
Administrator's Guide
Analyzing Workloads
The
triggering
condition
What to do about it
Where to get
more
information
See ALTER
small,
where
the
number
of bytes
of disk
storage
in use by
the
projectio
n (used_
bytes) is
more
than 10M
User with
dbadmin/
USER in the
pseudosupe
SQL
ruser role
Reference
has empty
Manual
password.
Table with
too many
partition
PARTITION BY
See ALTER
(new_partition_
count.
TABLE in the
SQL
Reference
Manual
LGE
threshold
setting is
lower than
the default
setting.
Administrator's Guide
Analyzing Workloads
The
triggering
condition
Tuple
Mover's
MoveOutInt
What to do about it
Where to get
more
information
See the
following
topics:
erval
parameter is
Monitoring
Events
set greater
than the
l
default
Tuple
Mover
setting.
Parameter
s
l
ACTIVE_
EVENTS
in the SQL
Reference
Manual
Tuple Mover Workload Analyzer does not trigger a tuning recommendation for
has been
disabled.
Too Many
ROS
containers
since the
last
mergeout
operation;
configuratio
n
parameters
are set
lower than
the default.
Administrator's Guide
Analyzing Workloads
The
triggering
condition
What to do about it
Too Many
ROS
containers
Where to get
more
information
since the
last
mergeout
operation;
the TM
Mergeout
service is
disabled.
Average
See the
CPU usage
following
exceeds
topics:
95% for 20
minutes.
Guidelines
for Setting
Pool
Parameter
s
ALTER
RESOUR
CE POOL
and Built-In
Pool
Configurati
on in the
SQL
Reference
Manual
Administrator's Guide
Analyzing Workloads
The
triggering
condition
What to do about it
Where to get
more
information
Partitioned
See ALTER
table data is
TABLE in the
not fully
SQL
reorganized
Reference
after
Manual
repartitionin
g.
Table has
multiple
See ALTER
TABLE in the
partition
SQL
keys within
Reference
the same
Manual
ROS
container.
Excessive
swap
activity;
average
memory
usage
exceeds
99% for 10
minutes.
A table does
See
not have
Incremental
any
Design.
Database
Designerdesigned
projections.
Administrator's Guide
Analyzing Workloads
The
triggering
condition
What to do about it
Where to get
more
information
Statistics
See the
public.<table>.<column>
following
topics:
l
falls outside
ANALYZ
E_
histogram).
STATISTI
CS() in the
SQL
Reference
Manual
l
Collecting
Statistics
in this
guide
Data
See
distribution
(s).
Designing for
in
Segmentation
segmented
projection is
skewed.
GROUP BY
spill event.
See
Incremental
Design.
See Also
l
ANALYZE_WORKLOAD
Administrator's Guide
Collecting Database Statistics
Plans involving different algorithms, such as hash join/merge join or group by hash/group by
pipelined operations
Without reasonably accurate statistics, the optimizer could choose a suboptimal projection or a
suboptimal join order for a query.
To understand how HP Vertica collects statistics, consider the following common scenario:
Scenario: You have a large table into which you load timestamp data on an ongoing basis (hourly,
daily, etc.). Then you run queries that select the recently-loaded rows from that table.
How the optimizer decides on a plan: You load days 1 through 15 into the table and run the
ANALYZE_STATISTICS function. When you next run a query that requests yesterday's data by
filtering on the timestamp column, the optimizer chooses an optimized plan. If on the next day, you
load day 16 data and run the same querybut you do not run ANALYZE_STATISTICS againthe
optimizer might conclude that the predicate results in only one row being returned because the date
range falls outside the histogram range and the data becomes stale. When the optimizer detects
that statistics are not current for a particular predicate (such as when a timestamp predicate is
outside a histogram's boundary), HP Vertica plans those queries using other considerations, such
as FK-PK constraints, when available.
Resolution: Run ANALYZE_STATISTICS() after you load new data, or day 16 in this example.
You can also look for statistics in the EXPLAIN plan; for example, when statistics are off outside a
histogram's boundaries, the EXPLAIN plan is annotated with a status. See Reacting to Stale
Statistics for details.
For information about how to...
See ...
ANALYZE_HISTOGRAM()
Administrator's Guide
Collecting Database Statistics
See ...
ANALYZE_STATISTICS()
ANALYZE_EXTERNAL_
ROW_COUNT
ANALYZE_WORKLOAD()
EXPORT_STATISTICS()
IMPORT_STATISTICS()
STATISTICS command
Remove statistics for a table and optionally specify the category
DROP_STATISTICS()
of statistics to drop
Remove row count statistics for an external table
DROP_EXTERNAL_
ROW_COUNT
V_
type, sort order, type of statistics, and the time at which columns
CATALOG.PROJECTION_
COLUMNS
Note: The HP Vertica query optimizer and the Database Designer both use the same set of
statistics. When there are ties, the optimizer chooses the projection that was created first.
Administrator's Guide
Collecting Database Statistics
ANALYZE_STATISTICS
ANALYZE_HISTOGRAM
ANALYZE_EXTERNAL_ROW_COUNT
You can also invoke this function manually using the DO_TM_TASK ('analyze_row_count')
function.
Using ANALYZE_STATISTICS
The ANALYZE_STATISTICS function computes full statistics on all objects, or on a per-table or
per-column basis. You must invoke this function explicitly.
The ANALYZE_STATISTICS() function:
Administrator's Guide
Collecting Database Statistics
Performs faster data sampling, which expedites the analysis of relatively small tables with a
large number of columns.
Lets you cancel the function mid analysis by issuing CTRL-C on vsql or invoking the
INTERRUPT_STATEMENT() function.
Records the last time statistics were run for a table so that subsequent calls to the function can
be optimized. See V_CATALOG.PROJECTION_COLUMNS for details.
Using ANALYZE_HISTOGRAM
ANALYZE_STATISTICS() is an alias for ANALYZE_HISTOGRAM(). The difference between the
two functions is that ANALYZE_HISTOGRAM lets you specify what percentage of data to read
from disk, so you have more control over deciding between sample accuracy and speed.
The ANALYZE_HISTOGRAM percent parameter specifies the amount of column data (from 1 - 100%)
that the function reads from disk. The default value is 10%, but you can use this parameter to
specify a smaller or larger percentage. Changing the percent value affects both the data collection
time and the histogram accuracy:
l
A smaller percent value reads less data off disk. Data collection is faster than with a larger
value, but histogram accuracy decreases, since the function samples less data.
A larger percent value reads more data off disk. Data collection takes longer than for a smaller
value, but the histogram accuracy increases, since the function samples a larger percentage of
collected data.
Regardless of the percent parameter value, ANALYZE_HISTOGRAM uses at most 128K (128,000)
rows of column data. This sample size (128K) consistently creates an accurate representation of
the sample data, even for columns with more than 1,000,000 rows. The function constructs a
histogram for a column by randomly selecting from the collected data. If the percent value you
specify equates to less than 128K rows, the function rounds the number to at least 128K rows, so
enough data to sample is read from disk. If a column has less than 128K rows, ANALYZE_
HISTOGRAM reads all rows from disk and analyzes the entire column.
Administrator's Guide
Collecting Database Statistics
Note: The sample data collected in a sample range is not indicative of how data should be
distributed.
Following are some examples of different size columns with the ANALYZE_HISTOGRAM percent
parameter set to the different values:
Column Size
Percent
<128K rows
20
All rows
Entire column
400,000 rows
10
128,000
128K rows
4,000,000 rows
10
400,000
128K rows
Note: If the column that you specify for the ANALYZE_HISTOGRAM function is first in a
projection's sort order, the function reads all data from disk to avoid a biased sample.
Using ANALYZE_EXTERNAL_ROW_COUNT
To calculate the exact number of rows in an external table, use the ANALYZE_EXTERNAL_ROW_
COUNT function.The Optimizer uses this count to optimize for queries that access external tables.
In particular, if an external table participates in a join, the Optimizer can now identify the smaller
table to be used as the inner input to the join, resulting in better query performance.
Examples
In this example, the ANALYZE_STATISTICS() function reads 10 percent of the disk data. This is
the static default value for this function. The function returns 0 for success:
=> SELECT ANALYZE_STATISTICS('shipping_dimension.shipping_key');
ANALYZE_STATISTICS
-------------------0
(1 row)
This example uses ANALYZE_HISTOGRAM () without specifying a percentage value. Since this
function has a default value of 10 percent, it returns the identical data as the ANALYZE_
STATISTICS() function, and returns 0 for success:
=> SELECT ANALYZE_HISTOGRAM('shipping_dimension.shipping_key');
ANALYZE_HISTOGRAM
-------------------
Administrator's Guide
Collecting Database Statistics
This example uses ANALYZE_HISTOGRAM (), specifying its percent parameter as 100, indicating
to read the entire disk to gather data. After the function performs a full column scan, it returns 0 for
success:
=> SELECT ANALYZE_HISTOGRAM('shipping_dimension.shipping_key', 100);
ANALYZE_HISTOGRAM
------------------0
The following query calculates the exact number of rows in the external table loader_rejects:
=> SELECT ANALYZE_EXTERNAL_ROW_COUNT('loader_rejects');
ANALYZE_EXTERNAL_ROW_COUNT
---------------------------0
Administrator's Guide
Collecting Database Statistics
The EXPLAIN plan is annotated with a status. See Reacting to Stale Statistics.
The last time ANALYZE_STATISTICS() was run for a table is recorded, so that subsequent
calls to the function are optimized. This is useful during the database design process because if
the Database Designer does not collect statistics when adding design tables, it generates a
warning indicating that statistics are old. You can then decide whether to run ANALYZE_
STATISTICS before you proceed with the design.
Administrator's Guide
Collecting Database Statistics
Notice that the statistics_type column returns NONE for all three columns in the trades table.
Also, there is no value in the statistics_updated_timestamp field because statistics have not
yet been run on this table.
-[RECORD 1 ]----------------+-------------------projection_id
| 45035996273718838
projection_name
| trades_p
projection_column_name
| stock
column_position
| 0
sort_position
| 0
column_id
| 45035996273718840
data_type
| char(5)
encoding_type
| RLE
access_rank
| 0
group_id
| 0
table_schema
| public
table_id
| 45035996273718836
table_name
| trades
table_column_id
| 45035996273718836-1
table_column_name
| stock
statistics_type
| NONE
statistics_updated_timestamp |
is_expression
| f
is_aggregate
| f
partition_by_position
|
order_by_type
|
column_expression
|
-[RECORD 2 ]----------------+-------------------projection_id
| 45035996273718838
projection_name
| trades_p
projection_column_name
| bid
column_position
| 1
sort_position
| 1
column_id
| 45035996273718842
data_type
| int
encoding_type
| DELTAVAL
access_rank
| 0
group_id
| 45035996273718844
table_schema
| public
table_id
| 45035996273718836
table_name
| trades
table_column_id
| 45035996273718836-2
table_column_name
| bid
statistics_type
| NONE
statistics_updated_timestamp |
is_expression
| f
is_aggregate
| f
partition_by_position
|
order_by_type
|
column_expression
|
-[RECORD 3 ]----------------+-------------------projection_id
| 45035996273718838
projection_name
| trades_p
projection_column_name
| ask
column_position
| 2
sort_position
|
column_id
| 45035996273718846
Administrator's Guide
Collecting Database Statistics
data_type
encoding_type
access_rank
group_id
table_schema
table_id
table_name
table_column_id
table_column_name
statistics_type
statistics_updated_timestamp
|
|
|
|
|
|
|
|
|
|
|
int
AUTO
0
45035996273718844
public
45035996273718836
trades
45035996273718836-3
ask
NONE
This time, statistics_type changes to FULL for the trades.stock column (representing full
statistics were run). The statistics_updated_timestamp column returns the time the stock
columns statistics were updated. The timestamp for the bid and ask columns have not changed
because statistics were not run on those columns. Also, the bid and ask columns changed from
NONE to ROWCOUNT because HP Vertica automatically updates ROWCOUNT statistics from
time to time. The statistics are created by looking at existing catalog metadata.
-[RECORD 1 ]----------------+-----------------------------projection_id
| 45035996273718838
projection_name
| trades_p
projection_column_name
| stock
column_position
| 0
sort_position
| 0
column_id
| 45035996273718840
data_type
| char(5)
encoding_type
| RLE
access_rank
| 0
group_id
| 0
table_schema
| public
table_id
| 45035996273718836
table_name
| trades
table_column_id
| 45035996273718836-1
table_column_name
| stock
statistics_type
| FULL
Administrator's Guide
Collecting Database Statistics
If you run statistics on the bid column and then query this system table again, only RECORD 2 is
updated:
Administrator's Guide
Collecting Database Statistics
Administrator's Guide
Collecting Database Statistics
projection_column_name
column_position
sort_position
column_id
data_type
encoding_type
access_rank
group_id
table_schema
table_id
table_name
table_column_id
table_column_name
statistics_type
statistics_updated_timestamp
is_expression
is_aggregate
partition_by_position
order_by_type
column_expression
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ask
2
45035996273718846
int
AUTO
0
45035996273718844
public
45035996273718836
trades
45035996273718836-3
ask
ROWCOUNT
2012-12-08 13:51:20.016475-05
f
f
You can query just the timestamp column to see when the columns were updated:
=> \x
Expanded display is off.
=> SELECT ANALYZE_STATISTICS('trades');
ANALYZE_STATISTICS
-------------------0
(1 row)
=> SELECT projection_column_name, statistics_type,
statistics_updated_timestamp
FROM PROJECTION_COLUMNS where table_name = 'trades';
projection_column_name | statistics_type | statistics_updated_timestamp
------------------------+-----------------+------------------------------stock
| FULL
| 2012-12-08 13:54:27.428622-05
bid
| FULL
| 2012-12-08 13:54:27.428632-05
ask
| FULL
| 2012-12-08 13:54:27.428639-05
(3 rows)
Administrator's Guide
Collecting Database Statistics
Ignores the stale statistics when it generates a query. Here, the optimizer executes queries
using other considerations, such as FK-PK constraints, when available.
The following EXPLAIN fragment shows that the predicate falls outside the histogram range:
| | +-- Outer -> STORAGE ACCESS for fact [Cost: 35, Rows: 1 (PREDICATE VALUE OUT-OFRANGE)]
You can get information about which table column has no statistics by querying a system table; for
example, view the timestamp for when statistics were last run by querying V_
CATALOG.PROJECTION_COLUMNS.
Example
First run full statistics on table 'trades':
=> SELECT ANALYZE_STATISTICS('trades'); ANALYZE_STATISTICS
-------------------0
(1 row)
Administrator's Guide
Collecting Database Statistics
See Also
l
Analyzing Workloads
Projections
Projection Columns
Administrator's Guide
Collecting Database Statistics
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
row count
(1 row)
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
analyze
for
for
for
for
for
for
for
for
for
for
for
for
for
for
for
for
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
projection
'product_dimension_DBD_3_seg_temp_init_temp_init'
'product_dimension_DBD_4_seg_temp_init_temp_init'
'promotion_dimension_DBD_5_seg_temp_init_temp_init'
'promotion_dimension_DBD_6_seg_temp_init_temp_init'
'shipping_dimension_DBD_13_seg_temp_init_temp_init'
'shipping_dimension_DBD_14_seg_temp_init_temp_init'
'vendor_dimension_DBD_10_seg_temp_init_temp_init'
'vendor_dimension_DBD_9_seg_temp_init_temp_init'
'warehouse_dimension_DBD_15_seg_temp_init_temp_init'
'warehouse_dimension_DBD_16_seg_temp_init_temp_init'
'store_dimension_DBD_19_seg_temp_init_temp_init'
'store_dimension_DBD_20_seg_temp_init_temp_init'
'store_orders_fact_DBD_23_seg_temp_init_temp_init'
'store_orders_fact_DBD_24_seg_temp_init_temp_init'
'store_sales_fact_DBD_21_seg_temp_init_temp_init'
'store_sales_fact_DBD_22_seg_temp_init_temp_init'
A new projection using a newly-loaded table is created and refreshed. Projections that have no
data never have full statistics. Use the PROJECTION_STORAGE system table to see if your
projection contains data.
New primary key values are added to tables with referential integrity constraints. When this
occurs, both the primary key and foreign key tables should be reanalyzed.
The relative table size, compared to tables it is being joined to, changes materially. For example,
a table becomes only five times larger than the other, when it was previously 50 times larger.
Administrator's Guide
Collecting Database Statistics
Tip: You can analyze statistics on a single table column, rather than on the entire table.
Running statistics on a single important column (such as the predicate column) is useful for
large tables, which could take a long time to compute. It's also a good idea to run statistics on a
column after you use ALTER TABLE to add or change a column.
Also consider using the ANALYZE_HISTOGRAM() function. Whereas ANALYZE_STATISTICS
uses a fixed value of 10 percent for the proportion of disk reads, ANALYZE_HISTOGRAM lets you
specify what percent of the disk to read. You can also diagnose and resolve many statistics-related
issues by calling the ANALYZE_WORKLOAD() function, which returns tuning recommendations.
If you update statistics and find that the query still performs poorly, run your query through the
Database Designer and choose incremental as the design type. See Incremental Design
Save Statistics
Once your system is running well, HP recommends that you save exported statistics for all tables.
In the unlikely scenario that statistics changes impact optimizer plans, particularly after an upgrade,
you can always revert back to the exported statistics.
The following topics in the SQL Reference Manual:
See Also
l
Analyzing Workloads
ANALYZE_STATISTICS
PROJECTION_COLUMNS
Administrator's Guide
Collecting Database Statistics
PROJECTION_STORAGE
PROJECTIONS
Administrator's Guide
Using Diagnostic Tools
Export a catalog.
The command returns the name of the installed package, which contains the version and build
numbers. The following example indicates that both HP Vertica 7.0.x and Management Console
7.0.x are running on the targeted host:
[dbadmin@myhost01 ~]$rpm -qa | grep vertica
vertica-7.0.0-0
vertica-console-7.0.0-0.x86_64
When you are logged in to your Vertica Analytic Database database, you can also run a query for
the version only, by running the following command:
dbadmin=> SELECT version();
version
--------------------------------------------Vertica Analytic Database v7.0.0-0
(1 row)
Administrator's Guide
Using Diagnostic Tools
Log files from the installation process, the database, and the administration tools (such as,
vertica.log, dbLog, admintools.log)
Error messages
Database design
Backup information
Requirements
scrutinize requires that a cluster be configured to support the Administration Tools utility. If
Administration Tools cannot run on the initiating host, then scrutinizecannot run on that host.
Further Information
l
Running scrutinize
Troubleshooting scrutinize
Running scrutinize
You can run scrutinize unqualified by any options:
$/opt/vertica/bin/scrutinize
When run without options, scrutinize collects a wide range of information from all cluster nodes.
It stores the results in a .tar file (VerticaScrutinizeNumericID.tar), with minimal effect on
database performance. scrutinizeoutput can help diagnose most issues and yet reduces upload
size by omitting fine-grained profiling data.
Administrator's Guide
Using Diagnostic Tools
scrutinize requires temporary disk space where it can collect data before posting the final
compressed (.tar) output. How much space depends on variables such as the size of the HP
Vertica log and extracted system tables, as well as user-specified options that limit the scope of
information collected. Before scrutinize runs, it verifies that the temporary directory contains at
least 1 GB of space; however, the amount that it actually needs can be much higher. By default,
scrutinize uses /opt/vertica/tmp as the temporary directory. To avoid problems, you can
create a temporary directory with as much space as you think scrutinize is likely to need. Then,
specify that scrutinize use this temporary directory through its --tmpdir option.
You can run scrutinize with one or more options that limit the scope of information that it collects.
For example, you can limit the time span of collected data, or collect data only from specific cluster
nodes. These options can help you narrow the focus of a collection to information that pertains to a
specific issue; they can also reduce output size and improve performance. For more information on
these options, see Limiting the Scope of scrutinize Data Collection.
Privileges
You must have database administrator (dbadmin) privileges to run scrutinize. If you run
scrutinizeas root when the dbadmin user exists, HP Vertica returns the following error:
[root@host01 ~]# /opt/vertica/bin/scrutinize
Root user is not allowed to use this tool.
Try again as the DB administrative user.
Command Options
Option
Description
--auth-upload=url
-A url
--url=url
file upload:
-u url
l
Administrator's Guide
Using Diagnostic Tools
Option
Description
--begin=timestamp
--end=timestamp
--begin specifies to collect log data from the specified begin time
to the specified end time or log end.
--end specifies to collect data from the specified begin time or log
start to the specified end time.
Examples
$scrutinize --begin='2014-01-01 00:00:00' --end='2014-01-07
23:59:59'
$scrutinize --begin='2014-01-01 00:00:00'
$scrutinize --end='2014-01-07 23:59:59'
$scrutinize --begin='2014-09-23 00:00:00' --end='1d''
--by-minute
--by-second
Administrator's Guide
Using Diagnostic Tools
Option
Description
--database=DB
-d DB
If you omit this option when these conditions are true, scrutinize
returns a message that asks you to specify a database.
Example
$scrutinize -d VMart
--diag-dump
--diagnostics
Limits the collection to log file data and output from commands that
are run against HP VerticaVertica and its host system. Use this
option to collect data to evaluate unexpected behavior in your HP
Vertica system.
--exclude-tasks=tasks
-X tasks
--get-files file-list
--help
-h
Administrator's Guide
Using Diagnostic Tools
Option
Description
--hosts=host-list
-n host-list
--include_
gzlogs=num-files
-z num-files
An integer specifies how many of the most recent rotated log files
to collect.
-s
Administrator's Guide
Using Diagnostic Tools
Option
Description
--log-limit limit
--message= message
-m message
reason.txt.
Messages are typically included when you upload scrutinize to HP
Vertica Customer Support.
You can set a message several ways:
l
--with-active-queries
--no-containers
--output_dir=path
-o path
Administrator's Guide
Using Diagnostic Tools
Option
Description
--password='password'
-P 'password'
PASSWORD.
Omitting this argument on a password-protected database returns a
warning. Enclose the password with single quotation marks (' ) to
protect passwords with special characters.
Use this option if the HP Vertica administrator account (default
dbadmin) has password authentication.
Example
$scrutinize -P '@passWord**'
$scrutinize --password='$password1*'
--tasks=tasks
-T tasks
--type=type
-t type
--tmpdir=path
--user=username
-U username
Administrator's Guide
Using Diagnostic Tools
Option
Description
--version
--vsql-off
-v
to the database. This option can help you deal with problems that
occur during an upgrade, and is typically used in the following cases:
l
-W
--prompt-password
You can use these options singly or in combination, to achieve the desired level of granularity.
--begin specifies to collect log data from the specified begin time to the specified end time or
log end.
Administrator's Guide
Using Diagnostic Tools
--end specifies to collect data from the specified begin time or log start to the specified end
time.
You can specify begin and end time arguments as offsets from the current time or as absolute
times:
Offset time format:
' DdHhMmSs'
For example:
'3d2h1m'
For example:
$scrutinize --begin='2014-01-01 00:00:00' --end='2014-01-07 23:59:59'
$scrutinize --begin='2014-01-01 00:00:00'
$scrutinize --end='2014-01-07 23:59:59'
$scrutinize --begin='2014-09-23 00:00:00' --end='1d''
Administrator's Guide
Using Diagnostic Tools
--include_gzlogs=num-files
-znum-files
--hosts=host-list
-nhost-list
was invoked.
$scrutinize --hosts=127.0.0.1,host_3,host_1
Limits the collection to database design, system tables, and Data Collector
tables. Use this option to collect data to analyze system performance.
Administrator's Guide
Using Diagnostic Tools
--diagnostics
Limits the collection to log file data and output from commands that are run
against HP Vertica and its host system. Use this option to collect data to
evaluate unexpected behavior in your HP Vertica system.
--type=type
-t type
--tasks=tasks
-T tasks
Specifies the type of diagnostics collection to perform, where type can be one of
the following arguments:
l
Excludes Query and SystemTable tasks, which are used to connect to the
database. This option can help you deal with problems that occur during an
upgrade, and is typically used in the following cases:
l
You haven't yet created a database but need help troubleshooting other
cluster issues.
Administrator's Guide
Using Diagnostic Tools
--exclude=tasks
-Xtasks
Excludes one or more types of tasks from the diagnostics collection, where
tasks is a comma-separated list of the tasks to exclude.
Note: This option is typically used only in consultation with your HP
Vertica Customer Support contact.
Specify the tasks to exclude with the following case-insensitive arguments :
l
File: Log files from the installation process, the database, and
Administration Tools, such as vertica.log, dbLog, and admintools.log
scrutinize Security
scrutinize supports password handling in the following ways:
l
The --password (-P) option sets the database password as an argument to the scrutinize
command.
Administrator's Guide
Using Diagnostic Tools
Use this option if the administrator account (default dbadmin) has password authentication.
l
The --prompt-password (-W) option invokes a DB Password prompt at the command line,
where users must enter their database password.
Troubleshooting scrutinize
The troubleshooting advice in this section can help you resolve common issues that you might
encounter when using scrutinize.
Administrator's Guide
Using Diagnostic Tools
Install the cURL program installed in the path for the database administrator user who is running
scrutinize.
Verify each node in the cluster can make an HTTP or FTP connection directly to the Internet.
A smaller, context file is posted first, which lets Customer Support review high-level
information.
The larger, more complete diagnostics collection downloads at the end of the run.
Authenticated Upload
You can use the --url or the --auth-upload option to upload scrutinize output to HP Vertica
Customer Support. Each option authenticates the upload in a different way:
l
The --url upload method requires the URLto include a user name and password.
When you use the --auth-upload option, scrutinize uploads your customer name from your
HP Vertica Enterprise Edition license. Customer Support uses this information to verify your
identity on receiving your uploaded file.
The --auth-upload argument requires a valid HP Vertica Enterprise Edition license. In both cases,
the upload uses a URLor FTP address supplied by HP Vertica Customer Support.
Upload Examples
The following example shows a generic upload that invokes the --url option to gather all
diagnostics on the cluster and posts the collected output to a support-provided URL:
Administrator's Guide
Using Diagnostic Tools
You can also use the authenticated upload method to gather the same information and your
customer name:
$scrutinize -U username -P 'password' --auth-upload "url"
Support-supplied case number and other issue-specific information, in order to help HP Vertica
Customer Support identify your case and analyze the problem.
If no message is specified, scrutinize generates the default message 'Unknown reason for
collection.'
You can set a message several ways:
l
Supply the path to a file enclosed by double quotation marks. scrutinize includes the file
contents in its output file. For example:
$scrutinize --message="/path/to/file"
The PROMPT keyword opens an input stream. scrutinize reads input until you type a period (.)
on a new line. This closes the input stream, and scrutinize writes the message to the
collected output. For example:
$scrutinize --message=PROMPT
Enter reason for collecting diagnostics; end with '.' on a line by itself:
Query performance degradation noticed around 9AM EST on Saturday
.
Vertica Scrutinize Report
-----------------------------
Administrator's Guide
Using Diagnostic Tools
Result Dir:
...
/home/dbadmin/VerticaScrutinize.20131126083311
Database Identifiers
scrutinize generates a unique identifier for the database that it analyzes. This identifier is based
on your HP Vertica license and the database creation time. scrutinize generates the same name
for a given database each time it collects data on that database. It displays this name near the end
of its standard outputin the following example, PURPLELEAD:
Vertica Scrutinize Report
----------------------------Result Dir:
/home/dbadmin/VerticaScrutinize.20140206144429
...
Gathered diagnostics for
Customer: Vertica Systems, Inc.
Database designation: PURPLE LEAD
Timestamp: 20140206144429
...
You can use this identifier when you interact with Customer Support. It is particularly useful in the
following cases:
l
Exporting a Catalog
When you export a catalog you can quickly move a catalog to another cluster. Exporting a catalog
transfers schemas, tables, constraints, projections, and views. System tables are not exported.
Exporting catalogs can also be useful for support purposes.
See the EXPORT_CATALOG function in the SQL Reference Manual for details.
Administrator's Guide
Using Diagnostic Tools
Syntax
/opt/vertica/scripts/collect_diag_dump.sh [command... ]
Parameters
command
-U
User name,
typically the
database
administrator
account,
dbadmin.
-w
Database
password.
-c
Includes a
compression
analysis,
resulting in a
longer script
execution
time.
Example
The following command runs the audit script with all arguments:
$/opt/vertica/scripts/collect_diag_dump.sh -U dbadmin -w password -c
Administrator's Guide
Understanding Query Plans
Cost
Cost is an estimate of the resources that the query plan will use for its execution strategy, such as
data distribution statistics, CPU, disk, memory, network, data segmentation across cluster nodes,
and so on. Although such resources correlate to query run time, they are not an estimate of run
time. For example, if Plan1 costs more than Plan2, the optimizer estimates that Plan2 will take less
time to run. Cost does not mean that if Pan1 costs two times more than Plan2, the optimizer
estimates that Plan2 will take half the time to run.
The optimizer does not compute cost across queries. For example, if you run a plan for Query1 and
a plan for Query2, a higher cost in Query2 does not indicate that Query2 will take longer than
Query1.
Statistics
Many important optimizer decisions rely on statistics, which the query optimizer uses to determine
the final plan to execute a query. Therefore, it is important that statistics be up to date. Without
reasonably accurate statistics, the optimizer could choose a suboptimal plan, which might affect
query performance.
HP Vertica uses the following statistics to calculate the lowest query plan cost and to create plan
candidates:
l
Administrator's Guide
Understanding Query Plans
The access path with the fewest expected I/O operations and lowest CPU, memory, and
network usage
Join types of differing algorithms (hash join/merge join or group by hash/group by pipelined)
Query predicates
Data re-distribution algorithms (broadcast and re-segmentation) across nodes in the cluster
HP Vertica provides hints about statistics through the query plan, which is annotated with a
statistics status. See Viewing the Statistics Query Plan Output.
See Also
Collecting Database Statistics
The output from a query plan is presented in a tree-like structure, where each step (path)
represents a single operation in the database that the optimizer uses for its execution strategy.
The following example output is based on the previous query:
------------------------ QUERY PLAN DESCRIPTION: ----------------------EXPLAIN SELECT
customer_name,
Administrator's Guide
Understanding Query Plans
customer_state
FROM customer_dimension
WHERE customer_state in ('MA','NH')
AND customer_gender = 'Male'
ORDER BY customer_name
LIMIT 10;
Access Path:
+-SELECT LIMIT 10 [Cost: 370, Rows: 10] (PATH ID: 0)
| Output Only: 10 tuples
| Execute on: Query Initiator
| +---> SORT [Cost: 370, Rows: 544] (PATH ID: 1)
| |
Order: customer_dimension.customer_name ASC
| |
Output Only: 10 tuples
| |
Execute on: Query Initiator
| | +---> STORAGE ACCESS for customer_dimension [Cost: 331, Rows: 544] (PATH ID: 2)
| | |
Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_
design_node0001
| | |
Materialize: customer_dimension.customer_state, customer_
dimension.customer_name
| | |
Filter: (customer_dimension.customer_gender = 'Male')
| | |
Filter: (customer_dimension.customer_state = ANY (ARRAY['MA', 'NH']))
| | |
Execute on: Query Initiator
See Also
l
EXPLAIN
QUERY_PLAN_PROFILES
Administrator's Guide
Understanding Query Plans
If you do not turn off the \o command, HP Vertica continues to save output to the file that you
specify. New output is appended to the previous output. The \o command captures all EXPLAIN
output in the file until you issue the \o command again.
View details about the projections and columns accessed by the query directly from the
EXPLAINoutput
After you select the database you want to use, you can view the EXPLAINoutput using
Management Console in either of two ways
l
Administrator's Guide
Understanding Query Plans
4. In the View Plan column, click Explainnext to the command for which you want to view the
query plan. Only certain queries, like SELECT,INSERT, DELETE, and UPDATE, have
EXPLAINplans.
5. In the Explain Plan window, click Explain and HP Vertica generates the EXPLAIN plan.
6. You can view the output in Path Information view or Tree Path view. To do so, click the
respective view buttons on the left of the output box.
Caution: Entering the word EXPLAINbefore the query results in a syntax error.
Administrator's Guide
Understanding Query Plans
You can also enter the transaction ID and statement ID or browse running or completed queries in
the Find a Query input window:
In the output window, you can perform the following tasks related to the query you entered:
l
Administrator's Guide
Understanding Query Plans
Path Information
Tree Path
Note: The Query Plan Drilldown and Profile Analysis output views are only available when
you run PROFILE.
You can change the EXPLAIN plan output using the icons on the bottom portion of the Explain
page.
The Path Information view displays the query plan path. You can expand or collapse the view to
see different levels of detail. To view metadata for a projection or a column, click the object name in
the path output. A pop-up window displays the metadata, if it is available.
Administrator's Guide
Understanding Query Plans
The
The Tree Path view details the query plan in the form of a tree. When you run EXPLAIN, the tree
view does not contain any metrics because the query has not yet executed.
Administrator's Guide
Understanding Query Plans
To collapse all the query paths in order to see a summary of each path in the EXPLAINoutput,
click Collapse All.
To expand all the query paths to display the path details, click Expand All.
To expand an individual query path so you can see details about that path, click the first line of
the path information. Click the first line again to collapse the path data.
Administrator's Guide
Understanding Query Plans
When you are done viewing the metadata, close the pop-up window.
EXPLAINSELECTcustomer_name, customer_state
FROM customer_dimension
WHERE customer_state IN ('MA','NH')
ORDERBY customer_name;
QUERY PLAN
---------------------------------------------------------------------------QUERY PLAN DESCRIPTION:
-----------------------------EXPLAIN SELECT customer_name, customer_state
FROM customer_dimension
WHERE customer_state IN ('MA','NH')
ORDER BY customer_name;
Access Path:
+-SORT [Cost: 2K, Rows: 50K (NO STATISTICS)] (PATH ID: 1)
| Order: customer_dimension.customer_name ASC
| Execute on: All Nodes
| +---> STORAGE ACCESS for customer_dimension [Cost: 235, Rows: 50K (NO STATISTICS)]
(PATH ID: 2)
| |
Projection: public.customer_dimension_b0
| |
Materialize: customer_dimension.customer_state, customer_dimension.customer_name
| |
Filter: (customer_dimension.customer_state = ANY (ARRAY['MA', 'NH']))
| |
Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
Incremental Design
The Path ID, an integer that links to error messages and profiling counters, making it easier for
you to troubleshoot performance issues
Administrator's Guide
Understanding Query Plans
Example
In the example EXPLAIN output that follows, the optimizer processes the query in three steps,
which are identified by path IDs:
1. Path ID 2: STORAGE ACCESS and FILTER
2. Path ID 1: SORT
3. Path ID 0: LIMIT
Note: A STORAGE ACCESS operation can scan more than the columns in the select list; for
example, columns referenced in WHERE clause, and so on.
The following table shows which portions of the query the three steps align with:
Path
STORAGE ACCESS
FILTER
Administrator's Guide
Understanding Query Plans
Path
SORT
ORDER BY customer_name
LIMIT
LIMIT 10;
New primary key values are added to tables with referential integrity constraints. Both the
primary key and foreign key tables should be reanalyzed.
Relative size of a table, compared to tables it is being joined to, has changed materially; for
example, one table is now only five times larger than the other table when previously it was 50
times larger.
There is a significant deviation in the distribution of data, which would necessitate recalculating
histograms. For example, there is an event that caused abnormally high levels of trading for a
particular stock.
HP Vertica provides hints about statistics through the query plan, which is annotated with a status
of either NO STATISTICS or STALE STATISTICS. For example, the following EXPLAIN fragment
indicates that histograms are unavailable (NOSTATISTICS):
| | +-- Outer -> STORAGE ACCESS for fact [Cost: 604, Rows: 10K (NO STATISTICS)]
The next EXPLAIN fragment indicates that the predicate has fallen outside the histogram range
(STALESTATISTICS):
| | +-- Outer -> STORAGE ACCESS for fact [Cost: 35, Rows: 1 (STALE STATISTICS)]
Administrator's Guide
Understanding Query Plans
Notes
l
You can resolve many issues related to statistics by calling the ANALYZE_STATISTICS()
function for the tables involved.
If you update statistics and find that the query still performs suboptimally, run your query through
Database Designer, choosing incremental design as the design type.
Projections that have no data never have full statistics. Query the PROJECTION_STORAGE
system table to see if your projection contains data. You can also query the PROJECTIONS
table, the HAS_STATISTICSfield.
Once your system is running well, save exported statistics for all tables. In the unlikely scenario
that statistics modifications impact optimizer plans, particularly after an upgrade, you can
always revert back to the exported statistics.
See Also
l
ANALYZE_STATISTICS
PROJECTION_COLUMNS
PROJECTION_STORAGE
PROJECTIONS
Administrator's Guide
Understanding Query Plans
The Row operator is the number of rows the optimizer estimates the query will return. Letters after
numbers refer to the units of measure (K=thousand, M=million, B=billion, T=trillion), so the output
for the following query indicates that the number of rows to return is 50 thousand.
=> EXPLAIN SELECT customer_gender FROM customer_dimension;
Access Path:
+-STORAGE ACCESS for customer_dimension [Cost: 17, Rows: 50K (3 RLE)] (PATH ID: 1)
| Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_design_
node0001
| Materialize: customer_dimension.customer_gender
| Execute on: Query Initiator
The reference to (3 RLE)in the STORAGEACCESSpath means that the optimizer estimates that
the storage access operator returns 50Krows. Because the column is run-length encoded (RLE),
the real number of RLE rows returned is only three rows:
l
Note: See Understanding Query Plans for additional information about how the optimizer
estimates cost.
Administrator's Guide
Understanding Query Plans
customer_name,
customer_state
FROM customer_dimension
WHERE customer_state in ('MA','NH')
AND customer_gender = 'Male'
ORDER BY customer_name
LIMIT 10;
Access Path:
+-SELECT LIMIT 10 [Cost: 370, Rows: 10] (PATH ID: 0)
| Output Only: 10 tuples
| Execute on: Query Initiator
| +---> SORT [Cost: 370, Rows: 544] (PATH ID: 1)
| |
Order: customer_dimension.customer_name ASC
| |
Output Only: 10 tuples
| |
Execute on: Query Initiator
| | +---> STORAGE ACCESS for customer_dimension [Cost: 331, Rows: 544] (PATH ID: 2)
| | |
Projection: public.customer_dimension_DBD_1_rep_vmart_vmart_node0001
| | |
Materialize: customer_dimension.customer_state, customer_dimension.customer_
name
| | |
Filter: (customer_dimension.customer_gender = 'Male')
| | |
Filter: (customer_dimension.customer_state = ANY (ARRAY['MA', 'NH']))
| | |
Execute on: Query Initiator
The query optimizer automatically picks the best projections, but without reasonably accurate
statistics, the optimizer could choose a suboptimal projection or join order for a query. For details,
see Collecting Statistics.
HP Vertica considers which projection to choose for a plan by considering the following aspects:
l
As HP Vertica scans the possibilities for each plan, projections with the higher initial costs could
end up in the final plan because they make joins cheaper. For example, a query can be answered
with many possible plans, which the optimizer considers before choosing one of them. For
efficiency, the optimizer uses sophisticated algorithms to prune intermediate partial plan fragments
with higher cost. The optimizer knows that intermediate plan fragments might initially look bad (due
to high storage access cost) but which produce excellent final plans due to other optimizations that
it allows.
If your statistics are up to date but the query still performs poorly, run the query through the
Database Designer. For details, see Incremental Design
Administrator's Guide
Understanding Query Plans
Tips
l
If you query an unsegmented projection by name, it changes the plan because then data is used
from one node only, where unsegmented projection names include a specific node name.
For optimal performance, write queries so the columns are sorted the same way that the
projection columns are sorted.
See Also
l
Physical Schema
In the following query, the T1 table is the left input because it is on the left side of the JOIN
keyword, and the T2 table is the right input, because it is on the right side of the JOIN keyword:
SELECT * FROM T1 JOIN T2 ON T1.x = T2.x;
Administrator's Guide
Understanding Query Plans
The following example shows a query and its plan for a full outer join:
=> EXPLAIN SELECT CD.annual_income,OSI.sale_date_key
-> FROM online_sales.online_sales_fact OSI
-> FULL OUTER JOIN customer_dimension CD ON CD.customer_key = OSI.customer_key;
Access Path:
+-JOIN HASH [FullOuter] [Cost: 18K, Rows: 5M] (PATH ID: 1) Outer (RESEGMENT) Inner
(FILTER)
| Join Cond: (CD.customer_key = OSI.customer_key)
| Execute on: All Nodes
| +-- Outer -> STORAGE ACCESS for OSI [Cost: 3K, Rows: 5M] (PATH ID: 2)
| |
Projection: online_sales.online_sales_fact_DBD_12_seg_vmartdb_design_vmartdb_
design
| |
Materialize: OSI.sale_date_key, OSI.customer_key
| |
Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for CD [Cost: 264, Rows: 50K] (PATH ID: 3)
| |
Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_design_
node0001
| |
Materialize: CD.annual_income, CD.customer_key
| |
Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
Tip: If you get a hash join when you are expecting a merge join, it means that at least one of
the projections is not sorted on the join column (for example, customer_key in the preceding
query). To facilitate a merge join, you might need to create different projections that are sorted
on the join columns.
In the next example, the optimizer chooses a merge join. The optimizer's first pass performs a
merge join because the inputs are presorted, and then it performs a hash join.
=> EXPLAIN SELECT count(*) FROM online_sales.online_sales_fact OSI
-> INNER JOIN customer_dimension CD ON CD.customer_key = OSI.customer_key
-> INNER JOIN product_dimension PD ON PD.product_key = OSI.product_key;
Access Path:
+-GROUPBY NOTHING [Cost: 8K, Rows: 1] (PATH ID: 1)
| Aggregates: count(*)
| Execute on: All Nodes
| +---> JOIN HASH [Cost: 7K, Rows: 5M] (PATH ID: 2)
| |
Join Cond: (PD.product_key = OSI.product_key)
| |
Materialize at Input: OSI.product_key
| |
Execute on: All Nodes
| | +-- Outer -> JOIN MERGEJOIN(inputs presorted) [Cost: 4K, Rows: 5M] (PATH ID: 3)
| | |
Join Cond: (CD.customer_key = OSI.customer_key)
| | |
Execute on: All Nodes
| | | +-- Outer -> STORAGE ACCESS for OSI [Cost: 3K, Rows: 5M] (PATH ID: 4)
| | | |
Projection: online_sales.online_sales_fact_DBD_12_seg_vmartdb_design_
vmartdb_design
| | | |
Materialize: OSI.customer_key
| | | |
Execute on: All Nodes
| | | +-- Inner -> STORAGE ACCESS for CD [Cost: 132, Rows: 50K] (PATH ID: 5)
| | | |
Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_
design_node0001
| | | |
Materialize: CD.customer_key
| | | |
Execute on: All Nodes
| | +-- Inner -> STORAGE ACCESS for PD [Cost: 152, Rows: 60K] (PATH ID: 6)
| | |
Projection: public.product_dimension_DBD_2_rep_vmartdb_design_vmartdb_design_
node0001
| | |
Materialize: PD.product_key
| | |
Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
Inequality joins
HP Vertica processes joins with equality predicates very efficiently. The EXPLAIN plan shows
equality join predicates as join condition (Join Cond).
=> EXPLAIN SELECT CD.annual_income, OSI.sale_date_key
-> FROM online_sales.online_sales_fact OSI
-> INNER JOIN customer_dimension CD
-> ON CD.customer_key = OSI.customer_key;
Access Path:
+-JOIN HASH [Cost: 4K, Rows: 5M] (PATH ID: 1)
| Join Cond: (CD.customer_key = OSI.customer_key)
| Materialize at Output: OSI.sale_date_key
| Execute on: All Nodes
| +-- Outer -> STORAGE ACCESS for OSI [Cost: 3K, Rows: 5M] (PATH ID: 2)
| |
Projection: online_sales.online_sales_fact_DBD_12_seg_vmartdb_design_vmartdb_
design
| |
Materialize: OSI.customer_key
| |
Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for CD [Cost: 264, Rows: 50K] (PATH ID: 3)
| |
Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_design_
node0001
| |
Materialize: CD.annual_income, CD.customer_key
| |
Execute on: All Nodes
However, inequality joins are treated like cross joins and can run less efficiently, which you can see
by the change in cost between the two queries:
=> EXPLAIN SELECT CD.annual_income, OSI.sale_date_key
-> FROM online_sales.online_sales_fact OSI
-> INNER JOIN customer_dimension CD
-> ON CD.customer_key < OSI.customer_key;
Access Path:
+-JOIN HASH [Cost: 98M, Rows: 5M] (PATH ID: 1)
| Join Filter: (CD.customer_key < OSI.customer_key)
| Materialize at Output: CD.annual_income
| Execute on: All Nodes
| +-- Outer -> STORAGE ACCESS for CD [Cost: 132, Rows: 50K] (PATH ID: 2)
| |
Projection: public.customer_dimension_DBD_1_rep_vmartdb_design_vmartdb_design_
node0001
| |
Materialize: CD.customer_key
| |
Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for OSI [Cost: 3K, Rows: 5M] (PATH ID: 3)
| |
Projection: online_sales.online_sales_fact_DBD_12_seg_vmartdb_design_vmartdb_
design
| |
Materialize: OSI.sale_date_key, OSI.customer_key
| |
Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
=> EXPLAIN SELECT * FROM hTicks h FULL OUTER JOIN aTicks a -> ON (h.time INTERPOLATE
PREVIOUS
Access Path:
+-JOIN (INTERPOLATED) [FullOuter] [Cost: 31, Rows: 4 (NO STATISTICS)] (PATH ID: 1)
Outer (SORT ON JOIN KEY) Inner (SORT ON JOIN KEY)
| Join Cond: (h."time" = a."time")
| Execute on: Query Initiator
| +-- Outer -> STORAGE ACCESS for h [Cost: 15, Rows: 4 (NO STATISTICS)] (PATH ID: 2)
| |
Projection: public.hTicks_node0004
| |
Materialize: h.stock, h."time", h.price
| |
Execute on: Query Initiator
| +-- Inner -> STORAGE ACCESS for a [Cost: 15, Rows: 4 (NO STATISTICS)] (PATH ID: 3)
| |
Projection: public.aTicks_node0004
| |
Materialize: a.stock, a."time", a.price
| |
Execute on: Query Initiator
EXPLAIN plans
Administrator's Guide
Understanding Query Plans
ID: 3)
| | | Projection: public.fact_super
| | | Materialize: fact.x
| | | Execute on: All Nodes
| | +-- Inner -> STORAGE ACCESS for dim [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH ID:
4)
| | | Projection: public.dim_super
| | | Materialize: dim.y
| | | Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for ext [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH ID:
5)
| | Projection: public.ext_super
| | Materialize: ext.z
| | Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
GROUPBYPIPELINED requires that inputs be presorted on the columns specified in the group,
which means that HP Vertica need only retain data in the current group in memory. GROUPBY
PIPELINED operations are preferred because they are generally faster and require less memory
than GROUPBY HASH. GROUPBYPIPELINED is especially useful for queries that process
large numbers of high-cardinality group by columns or DISTINCT aggregates.
GROUPBY HASH input is not sorted by the group columns, so HP Vertica builds a hash table
on those group columns in order to process the aggregates and group by expressions.
The optimizer chooses the faster GROUP BY PIPELINED over GROUP BY HASH if the certain
conditions are met.
Note: For details, see Avoiding GROUPBY HASH with Projection Design in the Analyzing
Data Guide.
For examples of queryplans that identify the GROUPBYalgorithm, see:
l
GROUPBYHASHQueryPlan Example
GROUPBYHASHQueryPlan Example
Here's an example of how GROUPBY HASHoperations look in EXPLAIN output.
=> EXPLAIN SELECT COUNT(DISTINCT annual_income)
FROM customer_dimension
WHERE customer_region='NorthWest';
The output shows that the optimizer chose the less efficient GROUPBY HASH path, which means
the projection was not presorted on the annual_income column. If such a projection is available,
the optimizer would choose the GROUPBYPIPELINEDalgorithm.
Access Path:
+-GROUPBY NOTHING [Cost: 256, Rows: 1 (NO STATISTICS)] (PATH ID: 1)
| Aggregates: count(DISTINCT customer_dimension.annual_income)
| +---> GROUPBY HASH (LOCAL RESEGMENT GROUPS) [Cost: 253, Rows: 10K (NO STATISTICS)]
(PATH ID: 2)
| |
Group By: customer_dimension.annual_income
| | +---> STORAGE ACCESS for customer_dimension [Cost: 227, Rows: 50K (NO STATISTICS)]
(PATH ID: 3)
| | |
Projection: public.customer_dimension_super
| | |
Materialize: customer_dimension.annual_income
| | |
Filter: (customer_dimension.customer_region = 'NorthWest'
...
Administrator's Guide
Understanding Query Plans
Similarly, the use of an equality predicate, such as in the following query, preserves GROUPBY
PIPELINED:
=> EXPLAIN SELECT COUNT(DISTINCT annual_income)
WHERE customer_gender = 'Female';
FROM customer_dimension
Tip: If EXPLAIN reports GROUPBY HASH, modify the projection design to force it to use
GROUPBY PIPELINED.
Administrator's Guide
Understanding Query Plans
CD.customer_name,
CD.customer_state,
AVG(CD.customer_age) AS avg_age,
COUNT(*) AS count
FROM customer_dimension CD
WHERE CD.customer_state in ('MA','NH')
AND CD.customer_gender = 'Male'
GROUP BY CD.customer_state, CD.customer_name
ORDER BY avg_age, customer_name;
Access Path:
+-SORT [Cost: 422, Rows: 544] (PATH ID: 1)
| Order: (<SVAR> / float8(<SVAR>)) ASC, CD.customer_name ASC
| Execute on: Query Initiator
| +---> GROUPBY HASH [Cost: 378, Rows: 544] (PATH ID: 2)
| |
Aggregates: sum_float(CD.customer_age), count(CD.customer_age), count(*)
| |
Group By: CD.customer_state, CD.customer_name
| |
Execute on: Query Initiator
| | +---> STORAGE ACCESS for CD [Cost: 372, Rows: 544] (PATH ID: 3)
| | |
Projection: public.customer_dimension_DBD_1_rep_vmart_vmart_node0001
| | |
Materialize: CD.customer_state, CD.customer_name, CD.customer_age
| | |
Filter: (CD.customer_gender = 'Male')
| | |
Filter: (CD.customer_state = ANY (ARRAY['MA', 'NH']))
| | |
Execute on: Query Initiator
If you change the sort order to descending, the change appears in the query plan:
EXPLAIN SELECT
CD.customer_name,
CD.customer_state,
AVG(CD.customer_age) AS avg_age,
COUNT(*) AS count
FROM customer_dimension CD
WHERE CD.customer_state in ('MA','NH')
AND CD.customer_gender = 'Male'
GROUP BY CD.customer_state, CD.customer_name
ORDER BY avg_age DESC, customer_name;
Access Path:
+-SORT [Cost: 422, Rows: 544] (PATH ID: 1)
| Order: (<SVAR> / float8(<SVAR>)) DESC, CD.customer_name ASC
| Execute on: Query Initiator
| +---> GROUPBY HASH [Cost: 378, Rows: 544] (PATH ID: 2)
| |
Aggregates: sum_float(CD.customer_age), count(CD.customer_age), count(*)
| |
Group By: CD.customer_state, CD.customer_name
| |
Execute on: Query Initiator
| | +---> STORAGE ACCESS for CD [Cost: 372, Rows: 544] (PATH ID: 3)
| | |
Projection: public.customer_dimension_DBD_1_rep_vmart_vmart_node0001
| | |
Materialize: CD.customer_state, CD.customer_name, CD.customer_age
| | |
Filter: (CD.customer_gender = 'Male')
| | |
Filter: (CD.customer_state = ANY (ARRAY['MA', 'NH']))
| | |
Execute on: Query Initiator
Administrator's Guide
Understanding Query Plans
One table is very small (usually the inner table) compared to the other
Outer join or subquery semantics require one side of the join to be replicated
Resegmentation takes an existing projection or intermediate relation and segments the data evenly
to each node in the cluster. At the end of the resegmentation operation, every row from the input
relation is on exactly one node. Resegmentation is the operation used most often for distributed
joins in HP Vertica if the data is not already segmented for local joins. See Using Identically
Segmented Projections in the Analyzing Data Guide.
Resegment example
=>CREATE TABLE T1 (a INT, b INT) SEGMENTED BY HASH(a) ALL NODES;
=> CREATE TABLE T2 (x INT, y INT) SEGMENTED BY HASH(x) ALL NODES;
Administrator's Guide
Understanding Query Plans
Broadcast example
=> EXPLAIN SELECT * FROM T1 LEFT JOIN T2 ON T1.a > T2.y;
Access Path:
+-JOIN HASH [LeftOuter] [Cost: 40K, Rows: 10K (NO STATISTICS)] (PATH ID: 1) Inner
(BROADCAST)
| Join Filter: (T1.a > T2.y)
| Materialize at Output: T1.b
| Execute on: All Nodes
| +-- Outer -> STORAGE ACCESS for T1 [Cost: 151, Rows: 10K (NO STATISTICS)] (PATH ID: 2)
| |
Projection: public.T1_b0
| |
Materialize: T1.a
| |
Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for T2 [Cost: 302, Rows: 10K (NO STATISTICS)] (PATH ID: 3)
| |
Projection: public.T2_b0
| |
Materialize: T2.x, T2.y
| |
Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
both groups share the same PARTITION BY deal_stage clause, the data does not need to be
resegmented between groups :
EXPLAIN SELECT
first_value(deal_size) OVER (PARTITION BY deal_stage
ORDER BY deal_size),
last_value(deal_size) OVER (PARTITION BY deal_stage
ORDER BY deal_size),
row_number() OVER (PARTITION BY deal_stage
ORDER BY largest_bill_amount)
FROM customer_dimension;
Access Path:
+-ANALYTICAL [Cost: 1K, Rows: 50K] (PATH ID: 1)
| Analytic Group
|
Functions: row_number()
|
Group Sort: customer_dimension.deal_stage ASC, customer_dimension.largest_bill_
amount ASC NULLS LAST
| Analytic Group
|
Functions: first_value(), last_value()
|
Group Filter: customer_dimension.deal_stage
|
Group Sort: customer_dimension.deal_stage ASC, customer_dimension.deal_size ASC
NULLS LAST
| Execute on: All Nodes
| +---> STORAGE ACCESS for customer_dimension [Cost: 263, Rows: 50K]
(PATH ID: 2)
| |
Projection: public.customer_dimension_DBD_1_rep_vmart_vmart_node0001
| |
Materialize: customer_dimension.largest_bill_amount,
customer_dimension.deal_stage, customer_dimension.deal_size
| |
Execute on: All Nodes
See Also
The Window OVER() Clause
Administrator's Guide
Understanding Query Plans
Node state
EXPLAIN output
All Nodes.
If fewer than 6 nodes are up, EXPLAINlists the
non-running nodes.
If the node list contains non-ephemeral nodes, the
Examples
In the following example, the down node is v_vmart_node0005, and the node v_vmart_node0006
will execute this run of the query.
=> EXPLAINSELECT * FROM test;
QUERY PLAN
----------------------------------------------------------------------------QUERY PLAN DESCRIPTION:
-----------------------------EXPLAINSELECT * FROM my1table;
Access Path:
+-STORAGE ACCESS for my1table [Cost: 10, Rows: 2] (PATH ID: 1)
| Projection: public.my1table_b0
| Materialize: my1table.c1, my1table.c2
| Execute on: All Except v_vmart_node0005
+-STORAGE ACCESS for my1table (REPLACEMENT FOR DOWN NODE) [Cost: 66, Rows: 2]
| Projection: public.my1table_b1
| Materialize: my1table.c1, my1table.c2
| Execute on: v_vmart_node0006
The All Permanent Nodes output in the following example fragment denotes that the node list is
for permanent (non-ephemeral) nodes only:
=> EXPLAIN SELECT * FROM my2table;
Access Path:
+-STORAGE ACCESS for my2table [Cost: 18, Rows:6 (NO STATISTICS)] (PATH ID: 1)
| Projection: public.my2tablee_b0
| Materialize: my2table.x, my2table.y, my2table.z
| Execute on: All Permanent Nodes
Administrator's Guide
Understanding Query Plans
When HP Vertica prepares a non-optimized execution plan for a MERGE statement, a RightOuter
path appears in the plan's textual output, such as in the following fragment.
...
Access Path: +-DML MERGE
| Target Projection: public.locations_b1
| Target Projection: public.locations_b0
| Target Prep:
| Execute on: All Nodes
| +---> JOIN MERGEJOIN(inputs presorted) [RightOuter] [Cost: 28, Rows: 3 (NO
STATISTICS)] (PATH ID: 1) Outer (RESEGMENT) Inner (RESEGMENT)
| |
Join Cond: (locations.user_id = VAL(2)) AND (locations.location_x = VAL(2)) AND
(locations.location_y = VAL(2))
| |
Execute on: All Nodes
| | +-- Outer -> STORAGE ACCESS for <No Alias> [Cost: 15, Rows: 2 (NO STATISTICS)] (PATH
ID: 2)
...
Administrator's Guide
Understanding Query Plans
Another way to see if HP Vertica will prepare an optimized query plan for MERGE is to grep for the
non-optimized MERGE path, 'RightOuter'. A null result after the EXPLAIN statement, such as in
the following example, means the MERGE statement will likely run with an optimized plan.
> \o | grep 'RightOuter'
> EXPLAIN MERGE INTO a USINGc ON a1 = c1
WHEN MATCHED THEN UPDATE SET a1 = c1, a2=c2,a3=c3
WHEN NOT MATCHED THEN INSERT (a1,a2,a3) VALUES(c1,c2,c3);
> \o
EXPLAIN output
Administrator's Guide
Understanding Query Plans
ERROR: Join inner did not fit in memory ((B x A)using B_sp and A_sp (PATH ID: 1))
Example
In the following series of commands, EXPLAIN returns the PATH ID for each plan, PROFILE
profiles the query, and the EXECUTION_ENGINE_PROFILES system table returns operating and
profiling counters, along with the PATH_ID that links back to EXPLAIN:
1. Run EXPLAIN <query>. The command in the following example output returns five paths,
each identified by a distinct PATH ID:
=> EXPLAIN SELECT * FROM fact JOIN dim ON x=y JOIN ext on y=z;
Access Path:
+-JOIN MERGEJOIN(inputs presorted) [Cost: 815, Rows: 10K (NO STATISTICS)] (PATH ID:
1)
| Join Cond: (dim.y = ext.z)
| Materialize at Output: fact.x
| Execute on: All Nodes
| +-- Outer -> JOIN MERGEJOIN(inputs presorted) [Cost: 408, Rows: 10K (NO
STATISTICS)] (PATH ID: 2)
| | Join Cond: (fact.x = dim.y)
| | Execute on: All Nodes
| | +-- Outer -> STORAGE ACCESS for fact [Cost: 202, Rows: 10K (NO STATISTICS)]
(PATH ID: 3)
| | | Projection: public.fact_super
| | | Materialize: fact.x
| | | Execute on: All Nodes
| | +-- Inner -> STORAGE ACCESS for dim [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH
ID: 4)
| | | Projection: public.dim_super
| | | Materialize: dim.y
| | | Execute on: All Nodes
| +-- Inner -> STORAGE ACCESS for ext [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH
ID: 5)
| | Projection: public.ext_super
| | Materialize: ext.z
| | Execute on: All Nodes
Administrator's Guide
Understanding Query Plans
PROFILES table.
=> PROFILE SELECT * FROM fact JOIN dim ON x=y JOIN ext on y=z;
NOTICE: Statement is being profiled.
HINT:
select * from v_monitor.execution_engine_profiles
where transaction_id=45035996273743212 and statement_id=2;
NOTICE: Initiator memory for query: [on pool sysquery: 16384 KB, minimum: 6020 KB]
NOTICE: Total memory required by query: [16384 KB]
3. Query the EXECUTION_ENGINE_PROFILES system table for the join operations and PATH
ID. The output refers to PATH IDs 1 and 2 in the EXPLAIN output.
=> SELECT node_name, operator_name, counter_name, path_id
FROM execution_engine_profiles
WHERE operator_name LIKE 'Join%' AND counter_name LIKE '%rows%';
node_name | operator_name
| counter_name
| path_id
-----------+------------------+-------------------------+--------e0
| JoinManyFewMerge | rows produced
| 1
e0
| JoinManyFewMerge | rle rows produced
| 1
e0
| JoinManyFewMerge | estimated rows produced | 1
e0
| JoinManyFewMerge | rows produced
| 2
e0
| JoinManyFewMerge | rle rows produced
| 2
e0
| JoinManyFewMerge | estimated rows produced | 2
e1
| JoinManyFewMerge | rows produced
| 1
e1
| JoinManyFewMerge | rle rows produced
| 1
e1
| JoinManyFewMerge | estimated rows produced | 1
e1
| JoinManyFewMerge | rows produced
| 2
e1
| JoinManyFewMerge | rle rows produced
| 2
e1
| JoinManyFewMerge | estimated rows produced | 2
initiator | JoinManyFewMerge | rows produced
| 1
initiator | JoinManyFewMerge | rle rows produced
| 1
initiator | JoinManyFewMerge | estimated rows produced | 1
initiator | JoinManyFewMerge | rows produced
| 2
initiator | JoinManyFewMerge | rle rows produced
| 2
initiator | JoinManyFewMerge | estimated rows produced | 2
(18 rows)
Memory used
Administrator's Guide
Understanding Query Plans
Administrator's Guide
Understanding Query Plans
Administrator's Guide
Profiling Database Performance
How the data has flowed through each operator at different points in time over the life of the
query
Real-time profiling is always "on"; you do not need to explicitly enable it.
Profile data is available for:
l
If profiling is not enabled, HP Vertica does not save the data in EXECUTION_ENGINE_
PROFILES, but some data is saved in QUERY_PROFILES., QUERY_PLAN_PROFILES, and
SESSION_PROFILES.
HP Vertica continues to make enhancements to the EXECUTION_ENGINE_PROFILES system
table and the Workload Analyzer, but the profiling methods described in this section are still a
valuable tool for gaining insight into what happens during execution time.
Administrator's Guide
Profiling Database Performance
See Also:
l
Profiling Functions
queryEnables profiling for general information about queries that ran, such as the query
strings used and the duration of queries.
Administrator's Guide
Profiling Database Performance
Examples
The following statement enables profiling for the execution run of each query:
=> SELECT ENABLE_PROFILING('ee');
ENABLE_PROFILING
---------------------EE Profiling Enabled
(1 row)
The following statement enables query profiling for all sessions and nodes:
=> ALTERDATABASE mydb SET GlobalSessionProfiling = 1;
sessionDisables profiling basic session parameters and lock time out data.
queryDisables profiling for general information about queries that ran, such as the query
strings used and the duration of queries.
Administrator's Guide
Profiling Database Performance
Example
The following command disables profiling on query execution runs:
=> SELECT DISABLE_PROFILING('ee');
DISABLE_PROFILING
----------------------EE Profiling Disabled
(1 row)
The following command disables query profiling for all sessions and nodes:
=> ALTERDATABASE mydb SET GlobalSessionProfiling = 0;
Rows produced
Administrator's Guide
Profiling Database Performance
Example
To retrieve a transaction ID or statement ID, use the following command:
=> SELECT transaction_id, statement_id FROM SYSTEM_SESSIONS;
To monitor the profiling counters, you can run a command like the following using a retrieved
transaction ID (a000000000027):
=> SELECT * FROM execution_engine_profiles
WHERE TO_HEX(transaction_id)='a000000000027'
AND counter_name = 'execution time (us)'
ORDER BY node_name, counter_value DESC;
In this example, find the operators with the largest execution time on each node:
=> SELECT node_name, operator_name, counter_value execution_time_us
FROM v_monitor.execution_engine_profiles
WHERE counter_name='execution time (us)'
ORDER BY node_name, counter_value DESC;
You can use the Linux watch command to monitor long-running queries with one-second updates:
WATCH -n 1 -d "vsql -c \"select node_name, operator_name,
counter_value execution_time_us ... \""
See Also
l
Administrator's Guide
Profiling Database Performance
System Table
Description
EXECUTION_ENGINE_PROFILES
QUERY_PLAN_PROFILES
QUERY_PROFILES
SESSION_PROFILES
View details about projection metadata, execution events, and optimizer events
Identify how much time was spent in each phase of query execution and which phases took the
most amount of time
After you select the database you want to use, you can view the profile data using Management
Consolein either of two ways:
l
Administrator's Guide
Profiling Database Performance
Caution: If you enter more than one query, Management Console profiles only the first
query.
2. To receive periodic updates about the query's progress and resources used, select the Enable
Monitoring check box. As a best practice, avoid specifying an interval time of less than 60
seconds because doing so may slow your query's progress.
3. Click the Profile button.
While HP Vertica is profiling the query, a Cancel Query button is enabled briefly, allowing you
to cancel the query and profiling task. If the Cancel Query button is disabled, that means
Management Console does not have the proper information to cancel the query or the query is
no longer running in the database.
Administrator's Guide
Profiling Database Performance
When processing completes, the profile data and metrics display below the text box. You can
view the output in Path Information view, Query Plan Drilldown view, Tree Path view , or
Profile Analysis view. To do so, click the respective view buttons on the left of the output box.
You can change the query profile output using the icons on the bottom portion of the Explain page.
The Path Information view displays the query plan path along with metric data. If you enable
profile monitoring, the data will update at the specified interval. To view metadata for a projection or
a column, click the object name in the path output. A pop-up window displays the metadata if it is
available.
Administrator's Guide
Profiling Database Performance
The Query Plan Drilldown view shows detailed counter information at the node and operator level.
For each path, the path number is listed along with statistical information on the node and operator
level. This view allows you to see which nodes are acting as outliers. Click on any of the bars to
expand details for that node.
The Tree Path details the query plan in the form of a tree. If you enable monitoring, the state of the
path blocks will change depending on whether the path is running, done, or has not started. Metric
information is displayed in each path block for the counters you specified in the Profile Settings.
Administrator's Guide
Profiling Database Performance
The Profile Analysis view allows you to identify any resource outliers. You can compare the
estimated rows produced count with the actual rows produced count, view execution time per path,
and identify memory usage per path.
Administrator's Guide
Profiling Database Performance
When you profile a query, you will also see a pie chart detailing the query phase duration. You can
also view projection metadata, execution events, and optimizer events by clicking on the
respective buttons next to the pie chart.
When a metric bar has decreased, a dark blue line indicates the previous metric.
When a metric bar has increased, a light blue bar indicates the added percentage. The previous
percentage appears as a dark blue bar.
When a metric has changed since the last interval, it is highlighted in yellow.
Administrator's Guide
Profiling Database Performance
To expand an individual query path so you can see details about that step in processing the
query, click the first line of the path information. Click the first line again to collapse the path
data.
For information about what the profile data means, see About Profile Data in Management Console.
Projection Metadata
Execution Events
Optimizer Events
Profile Metrics
Projection Metadata
To view projection metadata for a specific projection, click the projection name in the
EXPLAINoutput. Metadata for that projection opens in a pop-up window.
To view projection data for all projections accessed by that query, click the View Projection
Metadata button at the top of the Explain page. The metadata for all projections opens in a new
browser window.
Administrator's Guide
Profiling Database Performance
Note: If the View Projection Metadata button is not enabled, click Profile to retrieve the
profile data, including the projection metadata.
The projection metadata includes the following information:
l
Projection ID
Schema name
Sort columns
Owner name
To display a SQL script that can recreate the projection on a different cluster, click Click to get
export data. This script is identical to the output of the EXPORT_OBJECTS function. The
SQLscript opens in a pop-up window.
Copy and paste the command from this window, and click Close.
Plan
InitPlan
SerializePlan
Administrator's Guide
Profiling Database Performance
PopulateVirtualProjection
PreparePlan
CompilePlan
ExecutePlan
AbandonPlan
Hover over the slices on the pie chart or over the names of the phases in the box to get additional
information. You can see the approximate number of milliseconds(ms) and percentage used during
each phase.
Note: The time data in the profile metrics might not match the times in the query phase
duration. These times can differ because the query phase duration graph uses the longest
execution time for a given phase from all the nodes. Network latency can add more data, which
is not taken into account in these calculations.
Profile Metrics
In the Path Information view, the area to the right of each query path contains profile metrics for
that path.
l
DiskBytes of data accessed from disk by each query path. If none of the query paths
accessed the disk data, all the values are 0.
TimeNumber of milliseconds (ms) that the query path took to process on a given node, shown
on progress bars. The sum of this data does not match the total time required to execute the
query. This mismatch occurs because many tasks are executed in parallel on different nodes.
Hover over the progress bars to get more information, such as total bytes and percentages.
Administrator's Guide
Profiling Database Performance
Note: The time data in the profile metrics might not match the times in the Query Phase
Duration chart. These times can differ because the query phase duration graph uses the
longest execution time for a given phase from all the nodes. Network latency can add more
data, which is not taken into account in these calculations.
Execution Events
To help you monitor your database system, HP Vertica logs significant events that affect database
performance and functionality. Click View Execution Events to see information about the events
that took place while the query was executing.
If the View Execution Events button is not enabled, click Profile to retrieve the profile data,
including the execution events.
The arrows on the header of each column allow you to sort the table in ascending or descending
order of that column.
The execution events are described in the following table.
Event Characteristic
Details
Time
Node Name
Session ID
User ID
Request ID
Administrator's Guide
Profiling Database Performance
Event Type
NO HISTOGRAM
GROUP_BY_SPILLED
JOIN_SPILLED
PARTITIONS_ELIMINATED
MERGE_CONVERTED_TO_UNION
Event Description
Operator Name
DataSource
DataTarget
NetworkSend
NetworkRecv
StorageUnion
Values from the Operator name and Path IDcolumns let you tie a query
event back to a particular operator in the EXPLAIN plan. If the event did
not come from a specific operator, then this column is NULL.
Path ID
Event OID
Event Details
Suggested Action
Administrator's Guide
Profiling Database Performance
Optimizer Events
To help you monitor your database system, HP Vertica logs significant events that affect database
performance and functionality.Click View Optimizer Events to see a table of the events that took
place while the optimizer was planning the query.
If the View Optimizer Events button is not enabled, click Profile to retrieve the profile data,
including the optimizer events.
The arrows on the header of each column allow you to sort the table in ascending or descending
order of that column.
The following types of optimizer events may appear in the table:
Event characteristic
Details
Time
Node Name
Session ID
User ID
Request ID
Event Type
Event Description
Event OID
Event Details
Suggested Action
Administrator's Guide
Profiling Database Performance
While a query is executing, a notice and hint display in the terminal window. For example, the
preceding query returns the following:
NOTICE 4788: Statement is being profiled
HINT: Select * from v_monitor.execution_engine_profiles where transaction_
id=45035996273726389 and statement_id=10;
NOTICE 3557: Initiator memory for query: [on pool general: 732168 KB, minimum: 451218
KB]
NOTICE 5077: Total memory required by query: [732168 KB]
customer_name
| annual_income
-----------------+--------------James M. McNulty |
999979
Emily G. Vogel
|
999998
(2 rows)
Tip: Use the statement returned by the hint as a starting point for reviewing the query's
profiling data, such as to see what counters are available.
Administrator's Guide
Profiling Database Performance
Administrator's Guide
Profiling Database Performance
Administrator's Guide
Profiling Database Performance
To view the execution time and rows produced for a specific transaction and statement_id ranked
by execution time on each node:
=> SELECT * FROM v_demo.eeprof_execution_time_us_rank
WHERE transaction_id=45035996273709699
AND statement_id=1
ORDER BY transaction_id, statement_id, node_name, rk;
The Linux watch command is useful for long-running queries or long-running internal operations by
observing which part of the query plan is currently active:
=> watch -d -n 2 vsql -c \
=> SELECT * FROM v_demo.eeprof_execution_time_us_rank
WHERE transaction_id=45035996273709699 AND statement_id=1 AND rk<=5
ORDER BY transaction_id, statement_id, node_name, rk;\
This watch command executes the query every two seconds and highlights the differences
between successive updates.
Tip: Using watch is a convenient way to monitor the currently executing operators within the
Administrator's Guide
Profiling Database Performance
plan on each nodes in the HP Vertica cluster. watch is also a convenient way to monitor
workloads that might be unbalanced between nodes. For example, if some nodes become idle
while other nodes are still active. Such imbalances could be caused by data skews or by
hardware issues.
See Also
l
All characters must be alphanumeric with any number of underscores (_) and dollar signs ($)
Label syntax
Labels take the following form:
/*+label(label_name)*/
You can use optional spaces before and after the plus sign in label hints (between the /* and the +).
For example, HP Vertica accepts each of the following directives:
/*+label(label_name)*/
/* + label(label_name)*/
/*+ label(label_name)*/
/*+label(label_name) */
Example
Here's an example of some simple label hints and their associated queries:
SELECT /*+label(myselectquery)*/ COUNT(*) FROM t;
Administrator's Guide
Profiling Database Performance
INSERT
UPDATE
DELETE
SELECT
After you added a label to one or more queries, query the QUERY_PROFILES system table to see
which queries ran with your supplied labels.
The QUERY_PROFILES system table IDENTIFIER column returns the user-defined label that you
previously assigned to a query. Here's an example of the table's output using the query lables
=> SELECT identifier, query FROM query_profiles;
identifier | query
---------------+----------------------------------------------------------myselectquery | SELECT /*+label(myselectquery)*/ COUNT(*) FROM t;
myinsertquery | INSERT /*+label(myinsertquery)*/ INTO t VALUES(1);
myupdatequery | UPDATE /*+label(myupdatequery)*/ t SET a = 2 WHERE a = 1;
mydeletequery | DELETE /*+label(mydeletequery)*/ FROM t WHERE a = 1;
| SELECT identifier, query from query_profiles;
(5 rows)
Administrator's Guide
Profiling Database Performance
The same PATH ID is shared among EXPLAIN plans, join error messages, and EXECUTION_
ENGINE_PROFILES and QUERY_PLAN_PROFILES system tables, making it easy to quickly
trace issues to their root cause.
For each PATH ID, the QUERY_PLAN_PROFILES system table provides a high-level summary
of:
l
For example, you might observe that a GROUP BY HASH operation executed in 0.2 seconds using
100MB of memory.
Note: Real-time status is available in QUERY_PLAN_PROFILES system table output only; it
is not available in EXPLAIN plan output.
Administrator's Guide
Profiling Database Performance
See Also
l
Analyzing Workloads
Profiling counters are saved in the EXECUTION_ENGINE_PROFILES system table until the
storage quota has been exceeded.
Here's an example:
1. Profile the query to get the transaction_id and statement_id from from EXECUTION_
ENGINE_PROFILES; for example:
=> PROFILE SELECT * FROM t1 JOIN t2 ON t1.x = t2.y;
NOTICE 4788: Statement is being profiled
HINT: Select * from v_monitor.execution_engine_profiles where transaction_
id=45035996273955065 and statement_id=4;
NOTICE 3557: Initiator memory for query: [on pool general: 248544 KB, minimum:
248544 KB]
NOTICE 5077: Total memory required by query: [248544 KB]
x | y |
z
Administrator's Guide
Profiling Database Performance
---+---+------3 | 3 | three
(1 row)
Note: For best results, sort on the transaction_id, statement_id, path_id, and path_line_
index columns.
You can also use the Linux watch command to monitor long-running queries with two-second
updates. See Viewing Real-Time Profile Data.
Administrator's Guide
Profiling Database Performance
Example
The following series of commands creates a table for a long-running query and then runs the
QUERY_PLAN_PROFILES system table:
=> CREATE TABLE longq(x int);
CREATE TABLE
=> COPY longq FROM STDIN;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> 1
>> 2
>> 3
>> 4
>> 5
>> 6
>> 7
>> 8
>> 9
>> 10
>> \.
=> INSERT INTO longq SELECT f1.x+f2.x+f3.x+f4.x+f5.x+f6.x+f7.x
FROM longq f1
CROSS JOIN longq f2
CROSS JOIN longq f3
CROSS JOIN longq f4
CROSS JOIN longq f5
CROSS JOIN longq f6
CROSS JOIN longq f7;
OUTPUT
---------10000000
(1 row)
=> COMMIT;
COMMIT
Suppress query output on the terminal window by using the vsql \o command:
=> \o /home/dbadmin/longQprof
Administrator's Guide
Profiling Database Performance
(1 row)
Turn off the \o command so that HP Vertica continues to save query plan information to the file you
specified. Alternatively, leave it on and examine the file after you query the QUERY_PLAN_
PROFILES system table.
=> \o
Sort the system table output by transaction_id, statement_id, path_id, and path_line_
index, as in this example:
=> SELECT ... FROM query_plan_profiles
WHERE ...
ORDER BY transaction_id, statement_id, path_id, path_line_index;
Consider using column aliases to decrease column width, as in the following sample query:
=> SELECT statement_id AS sid, path_id AS id, path_line_index AS order,
is_started AS start, is_completed AS end, is_executing AS exe,
running_time AS run, memory_allocated_bytes AS mem,
read_from_disk_bytes AS read, received_bytes AS rec,
sent_bytes AS sent, FROM query_plan_profiles
Administrator's Guide
Profiling Database Performance
Suppress output from the screen and save profiling data to a file by using the vsql \o command.
For example:
a. Turn on the \o command.
=> \o /home/dbadmin/long-queries
CLEAR_PROFILING() clears profiled data from memory. For example, the following command
clears profiling for general query-run information, such as the query strings used and the duration
of queries.
=> SELECT CLEAR_PROFILING('query');
CLEAR_DATA_COLLECTOR() clears all memory and disk records on the Data Collector tables
and functions and resets collection statistics in the V_MONITOR.DATA_COLLECTOR system
table.
FLUSH_DATA_COLLECTOR() waits until memory logs are moved to disk and then flushes the
Data Collector, synchronizing the DataCollector log with the disk storage.
Administrator's Guide
Profiling Database Performance
ANALYZE_STATISTICS uses a fixed-size statistical data sampling (10% per disk). This
function returns results quickly, but it is less accurate than using ANALYZE_HISTOGRAM to
get a larger sampling of disk data.
ANALYZE_HISTOGRAM takes a user-specified percentage of disk data sampling (from 1 100). If you analyze more than 10 percent data per disk, this function returns more accurate
data than ANALYZE_STATISTICS, but it takes proportionately longer to return statistics.
You can also run your query through the Database Designer. See Incremental Design.
Administrator's Guide
About Locales
About Locales
HP Vertica supports the following internationalization features:
Locales
The locale is a parameter that defines the user's language, country, and any special variant
preferences, such as collation. HP Vertica uses the locale to determine the behavior of certain
string functions. The locale also determines the collation for various SQL commands that require
ordering and comparison, such as GROUP BY, ORDER BY, joins, and the analytic ORDER BY
clause.
By default, the locale for your HP Vertica database is en_US@collation=binary (English US). You
can define a new default locale that is used for all sessions on the database. You can also override
the locale for individual sessions. However, projections are always collated using the default en_
US@collation=binary collation, regardless of the session collation. Any locale-specific collation is
applied at query time.
You can set the locale through ODBC, JDBC, and ADO.net.
See the following topics in the Administrator's Guide
Administrator's Guide
About Locales
Supported Locales
Notes
l
The maximum length parameter for VARCHAR and CHAR data type refers to the number of
octets (bytes) that can be stored in that field and not number of characters. When using multibyte UTF-8 characters, the fields must be sized to accommodate from 1 to 4 bytes per
character, depending on the data.
When the locale is non-binary, the COLLATION function is used to transform the input to a
binary string which sorts in the proper order.
This transformation increases the number of bytes required for the input according to this
formula:
result_column_width = input_octet_width * CollationExpansion + 4
CollationExpansion defaults to 5.
BTRIM removes the longest string consisting only of specified characters from the start and end
of a string.
GREATESTB returns its greatest argument, using binary ordering, not UTF-8 character
ordering.
INITCAP capitalizes first letter of each alphanumeric word and puts the rest in lowercase.
Administrator's Guide
About Locales
INSTR searches string for substring and returns an integer indicating the position of the
character in string that is the first character of this occurrence.
LEASTB returns its least argument, using binary ordering, not UTF-8 character ordering.
LEFT returns the specified characters from the left side of a string.
LENGTH takes one argument as an input and returns returns an integer value representing the
number of characters in a string.
LTRIM returns a VARCHAR value representing a string with leading blanks removed from the
left side (beginning).
OVERLAY returns a VARCHAR value representing a string having had a substring replaced by
another string.
OVERLAYB returns an octet value representing a string having had a substring replaced by
another string.
REPLACE replaces all occurrences of characters in a string with another set of characters.
SUBSTRING given a value, a position, and an optional length, returns a value representing a
substring of the specified string at the given position.
UPPER returns a VARCHAR value containing the argument converted to uppercase letters.
Administrator's Guide
About Locales
LOWER returns a VARCHAR value containing the argument converted to lowercase letters.
UPPER returns a VARCHAR value containing the argument converted to uppercase letters.
INITCAP capitalizes first letter of each alphanumeric word and puts the rest in lowercase.
INSTR searches string for substring and returns an integer indicating the position of the
character in string that is the first character of this occurrence.
SPLIT_PART splits string on the delimiter and returns the location of the beginning of the given
field (counting from one).
POSITION returns an integer value representing the character location of a specified substring
with a string (counting from one).
STRPOS returns an integer value representing the character location of a specified substring
within a string (counting from one).
Administrator's Guide
About Locales
Locale Specification
The locale is a parameter that defines the user's language, country, and any special variant
preferences, such as collation. HP Vertica uses the locale to determine the behavior of certain
string functions. The locale also determines the collation for various SQL commands that require
ordering and comparison, such as GROUP BY, ORDER BY, joins, and the analytic ORDER BY
clause.
By default, the locale for your HP Vertica database is en_US@collation=binary (English US). You
can define a new default locale that is used for all sessions on the database. You can also override
the locale for individual sessions. However, projections are always collated using the default en_
US@collation=binary collation, regardless of the session collation. Any locale-specific collation is
applied at query time.
You can set the locale through ODBC, JDBC, and ADO.net.
HP Vertica locale specifications follow a subset of the Unicode LDML standard as implemented by
the ICU library.
Locales are specified using long or short forms.
Long Form
The long form uses full keyname pair/value names.
Syntax
[language][_script][_country][_variant][@keyword=type[;keyword=type]...]
Parameters
language
A two- or three-letter lowercase code for a particular language. For example, Spanish
is es English is en and French is fr. The two-letter language code uses the ISO-639
standard.
_script
An optional four-letter script code that follows the language code. If specified, it should
be a valid script code as listed on the Unicode ISO 15924 Registry.
Administrator's Guide
About Locales
_country
_variant
Differences may also appear in language conventions used within the same country.
For example, the Euro currency is used in several European countries while the
individual country's currency is still in circulation. To handle variations inside a
language and country pair, add a third code, the variant code. The variant code is
arbitrary and completely application-specific. ICU adds _EURO to its locale
designations for locales that support the Euro currency. Variants can have any number
of underscored key words. For example, EURO_WIN is a variant for the Euro currency
on a Windows computer.
Another use of the variant code is to designate the Collation (sorting order) of a locale.
For instance, the es__TRADITIONAL locale uses the traditional sorting order which is
different from the default modern sorting of Spanish.
Administrator's Guide
About Locales
keyword
Use optional keywords and their values to specify collation order and currency instead
of variants (as desired). If used, keywords must be unique, but their order is not
significant. If a keyword is unknown or unsupported, an error occurs. Keywords and
values are not case sensitive.
HP Vertica supports the following keywords:
Administrator's Guide
About Locales
Administrator's Guide
About Locales
Parameter
Short
form Description
colstrength
colAlternate
Administrator's Guide
About Locales
Parameter
Short
form Description
colBackwards
colNormalization
colCaseLevel
colCaseFirst
If set to upper (short form U), causes upper case to sort before
lower case. If set to lower (short form L)), lower case sorts
before upper case. This is useful for locales that have already
supported ordering but require different order of cases. It
affects case and tertiary levels.
If set to off (short form X), tertiary weights are not affected.
colHiraganaQuarternary
Administrator's Guide
About Locales
Parameter
Short
form Description
colNumeric
variableTop
Sets the default value for the variable top. All code points with
primary weights less than or equal to the variable top will be
considered variable, and are affected by the alternate handling.
For example, the following command sets variableTop to be
HYPHEN ('\u2010')
=> \locale en_US@colalternate=shifted;variabletop=u2010
Usage Considerations
l
Locale specification strings are case insensitive. The following are all equivalent: en_us, EN_US,
and En_uS.
The ICU library works by adding options, such as S=1 separately after the long-form locale has
been accepted. HP Vertica has extended its long-form processing to accept options as
keywords, as suggested by the Unicode Consortium.
Incorrect locale strings are accepted if the prefix can be resolved to a known locale version.
For example, the following works because the language can be resolved:
=> \locale en_XX
INFO 2567: Canonical locale: 'en_XX'
Standard collation: 'LEN'
English (XX)
The following does not work because the language cannot be resolved:
=> \locale xx_XX
xx_XX: invalid locale identifier
Administrator's Guide
About Locales
Invalid values of the collation keyword and its synonyms do not cause an error. For example, the
following does not generate an error. It simply ignores the invalid value:
=> \locale en_GB@collation=xyz
INFO 2567: Canonical locale: 'en_GB@collation=xyz'
Standard collation: 'LEN'
English (United Kingdom, collation=xyz)
POSIX-type locales, such as en_US.UTF-8 work to some extent in that the encoding part "UTF8" is ignored.
HP Vertica 4.0 uses the icu4c-4_2_1 library to support basic locale/collation processing with
some extensions as noted here. This does not yet meet the current standard for locale
processing: http://tools.ietf.org/html/rfc5646.
Examples
The following specifies a locale for English as used in the United States:
en_US
The following specifies a locale for English as used in the United Kingdom:
en_GB
The following specifies a locale for German as used in Deutschland and uses phonebook-style
collation.
=> \locale de_DE@collation=phonebook
INFO 2567: Canonical locale: 'de_DE@collation=phonebook'
Standard collation: 'KPHONEBOOK_LDE'
German (Germany, collation=Phonebook Sort Order)
Deutsch (Deutschland, Sortierung=Telefonbuch-Sortierregeln)
The following specifies a locale for German as used in Deutschland. It uses phonebook-style
collation with a strength of secondary.
=> \locale de_DE@collation=phonebook;colStrength=secondary
INFO 2567: Canonical locale: 'de_DE@collation=phonebook'
Standard collation: 'KPHONEBOOK_LDE_S2'
German (Germany, collation=Phonebook Sort Order)
Administrator's Guide
About Locales
Short Form
HP Vertica accepts locales in short form. You can use the short form to specify the locale and
keyname pair/value names.
The following example specifies a locale for German as used in Deutschland, and it uses
phonebook-style collation.
\locale LDE_KPHONEBOOKINFO: Locale: 'de@collation=phonebook'
INFO: German (collation=Phonebook Sort Order)
INFO: Deutsch (Sortierung=Telefonbuch-Sortierregeln)
INFO: Short form: 'KPHONEBOOK_LDE'
The following example specifies a locale for German as used in Deutschland. It uses phonebookstyle collation with a strength of secondary (see "Collation Keyword Parameters" in Long Form).
\locale LDE_KPHONEBOOK_S2INFO: Locale: 'de@collation=phonebook'
INFO: German (collation=Phonebook Sort Order)
INFO: Deutsch (Sortierung=Telefonbuch-Sortierregeln)
INFO: Short form: 'KPHONEBOOK_LDE_S2'
Administrator's Guide
About Locales
Supported Locales
The following are the supported locale strings for HP Vertica. Each locale can optionally have a list
of key/value pairs (see Long Form).
Locale Name
af
Afrikaans
af_NA
Afrikaans
Namibian Afrikaans
af_ZA
Afrikaans
South Africa
am
Ethiopic
am_ET
Ethiopic
ar
Arabic
ar_AE
Arabic
ar_BH
Arabic
Bahrain
ar_DZ
Arabic
Algeria
ar_EG
Arabic
Egypt
ar_IQ
Arabic
Iraq
ar_JO
Arabic
Jordan
ar_KW
Arabic
Kuwait
ar_LB
Arabic
Lebanon
ar_LY
Arabic
Libya
ar_MA
Arabic
Morocco
ar_OM
Arabic
Oman
ar_QA
Arabic
Qatar
ar_SA
Arabic
Saudi Arabia
ar_SD
Arabic
Sudan
Ethiopia
Administrator's Guide
About Locales
Locale Name
ar_SY
Arabic
Syria
ar_TN
Arabic
Tunisia
ar_YE
Arabic
Yemen
as
Assamese
as_IN
Assamese
az
Azerbaijani
az_Cyrl
Azerbaijani
Cyrillic
az_Cyrl_AZ
Azerbaijani
Azerbaijan Cyrillic
az_Latn
Azerbaijani
Latin
az_Latn_AZ
Azerbaijani
Azerbaijan Latin
be
Belarusian
be_BY
Belarusian
bg
Bulgarian
bg_BG
Bulgarian
bn
Bengali
bn_BD
Bengali
Bangladesh
bn_IN
Bengali
India
bo
Tibetan
bo_CN
Tibetan
PR China
bo_IN
Tibetan
India
ca
Catalan
ca_ES
Catalan
cs
Czech
cs_CZ
Czech
India
Belarus
Bulgaria
Spain
Czech Republic
Administrator's Guide
About Locales
Locale Name
cy
Welsh
cy_GB
Welsh
da
Danish
da_DK
Danish
de
German
de_AT
German
Austria
de_BE
German
Belgium
de_CH
German
Switzerland
de_DE
German
Germany
de_LI
German
Liechtenstein
de_LU
German
Luxembourg
el
Greek
el_CY
Greek
Cyprus
el_GR
Greek
Greece
en
English
en_AU
English
Australia
en_BE
English
Belgium
en_BW
English
Botswana
en_BZ
English
Belize
en_CA
English
Canada
en_GB
English
United Kingdom
en_HK
English
en_IE
English
Ireland
en_IN
English
India
United Kingdom
Denmark
Administrator's Guide
About Locales
Locale Name
en_JM
English
Jamaica
en_MH
English
Marshall Islands
en_MT
English
Malta
en_NA
English
Namibia
en_NZ
English
New Zealand
en_PH
English
Philippines
en_PK
English
Pakistan
en_SG
English
Singapore
en_TT
English
en_US_POSIX
English
en_VI
English
en_ZA
English
en_ZW
English
Zimbabwe
eo
Esperanto
es
Spanish
es_AR
Spanish
Argentina
es_BO
Spanish
Bolivia
es_CL
Spanish
Chile
es_CO
Spanish
Columbia
es_CR
Spanish
Costa Rica
es_DO
Spanish
Dominican Republic
es_EC
Spanish
Ecuador
es_ES
Spanish
Spain
es_GT
Spanish
Guatemala
Administrator's Guide
About Locales
Locale Name
es_HN
Spanish
Honduras
es_MX
Spanish
Mexico
es_NI
Spanish
Nicaragua
es_PA
Spanish
Panama
es_PE
Spanish
Peru
es_PR
Spanish
Puerto Rico
es_PY
Spanish
Paraguay
es_SV
Spanish
El Salvador
es_US
Spanish
United States
es_UY
Spanish
Uruguay
es_VE
Spanish
Venezuela
et
Estonian
et_EE
Estonian
Estonia
eu
Basque
Spain
eu_ES
Basque
Spain
fa
Persian
fa_AF
Persian
Afghanistan
fa_IR
Persian
Iran
fi
Finnish
fi_FI
Finnish
fo
Faroese
fo_FO
Faroese
fr
French
fr_BE
French
Finland
Faroe Islands
Belgium
Administrator's Guide
About Locales
Locale Name
fr_CA
French
Canada
fr_CH
French
Switzerland
fr_FR
French
France
fr_LU
French
Luxembourg
fr_MC
French
Monaco
fr_SN
French
Senegal
ga
Gaelic
ga_IE
Gaelic
gl
Gallegan
gl_ES
Gallegan
gsw
German
gsw_CH
German
gu
Gujurati
gu_IN
Gujurati
gv
Manx
gv_GB
Manx
ha
Hausa
ha_Latn
Hausa
Latin
ha_Latn_GH
Hausa
Ghana (Latin)
ha_Latn_NE
Hausa
Niger (Latin)
ha_Latn_NG
Hausa
Nigeria (Latin)
haw
Hawaiian
Hawaiian
haw_US
Hawaiian
United States
he
Hebrew
Ireland
Spain
Switzerland
India
United Kingdom
Administrator's Guide
About Locales
Locale Name
he_IL
Hebrew
hi
Hindi
hi_IN
Hindi
hr
Croation
hr_HR
Croation
hu
Hungarian
hu_HU
Hungarian
hy
Armenian
hy_AM
Armenian
Armenia
hy_AM_REVISED
Armenian
Revised Armenia
id
Indonesian
id_ID
Indonesian
ii
Sichuan
ii_CN
Sichuan
is
Icelandic
is_IS
Icelandic
it
Italian
it_CH
Italian
Switzerland
it_IT
Italian
Italy
ja
Japanese
ja_JP
Japanese
ka
Georgian
ka_GE
Georgian
kk
Kazakh
Israel
India
Croatia
Hungary
Indonesia
Yi
Iceland
Japan
Georgia
Administrator's Guide
About Locales
Locale Name
kk_Cyrl
Kazakh
Cyrillic
kk_Cyrl_KZ
Kazakh
Kazakhstan (Cyrillic)
kl
Kalaallisut
kl_GL
Kalaallisut
km
Khmer
km_KH
Khmer
kn
Kannada
kn-IN
Kannada
ko
Korean
ko_KR
Korean
kok
Konkani
kok_IN
Konkani
kw
Cornish
kw_GB
Cornish
lt
Lithuanian
lt_LT
Lithuanian
lv
Latvian
lv_LV
Latvian
mk
Macedonian
mk_MK
Macedonian
ml
Malayalam
ml_IN
Malayalam
mr
Marathi
mr_IN
Marathi
Greenland
Cambodia
India
Korea
India
United Kingdom
Lithuania
Latvia
Macedonia
India
India
Administrator's Guide
About Locales
Locale Name
ms
Malay
ms_BN
Malay
Brunei
ms_MY
Malay
Malaysia
mt
Maltese
mt_MT
Maltese
nb
Norwegian Bokml
nb_NO
Norwegian Bokml
ne
Nepali
ne_IN
Nepali
India
ne_NP
Nepali
Nepal
nl
Dutch
nl_BE
Dutch
Belgium
nl_NL
Dutch
Netherlands
nn
Norwegian nynorsk
nn_NO
Norwegian nynorsk
om
Oromo
om_ET
Oromo
Ethiopia
om_KE
Oromo
Kenya
or
Oriya
or_IN
Oriya
pa
Punjabi
pa_Arab
Punjabi
Arabic
pa_Arab_PK
Punjabi
Pakistan (Arabic)
pa_Guru
Punjabi
Gurmukhi
Malta
Norway
Norway
India
Administrator's Guide
About Locales
Locale Name
pa_Guru_IN
Punjabi
pl
Polish
pl_PL
Polish
ps
Pashto
ps_AF
Pashto
pt
Portuguese
pt_BR
Portuguese
Brazil
pt_PT
Portuguese
Portugal
ro
Romanian
ro_MD
Romanian
Moldavia
ro_RO
Romanian
Romania
ru
Russian
ru_RU
Russian
Russia
ru_UA
Russian
Ukraine
si
Sinhala
si_LK
Sinhala
sk
Slovak
sk_SK
Slovak
sl
Slovenian
sl_SL
Slovenian
so
Somali
so_DJ
Somali
Djibouti
so_ET
Somali
Ethiopia
so_KE
Somali
Kenya
India (Gurmukhi)
Poland
Afghanistan
Sri Lanka
Slovakia
Slovenia
Administrator's Guide
About Locales
Locale Name
so_SO
Somali
sq
Albanian
sq_AL
Albanian
sr
Serbian
sr_Cyrl
Serbian
Cyrillic
sr_Cyrl_BA
Serbian
sr_Cyrl_ME
Serbian
Montenegro (Cyrillic)
sr_Cyrl_RS
Serbian
Serbia (Cyrillic)
sr_Latn
Serbian
Latin
sr_Latn_BA
Serbian
sr_Latn_ME
Serbian
Montenegro (Latin)
sr_Latn_RS
Serbian
Serbia (Latin)
sv
Swedish
sv_FI
Swedish
Finland
sv_SE
Swedish
Sweden
sw
Swahili
sw_KE
Swahili
Kenya
sw_TZ
Swahili
Tanzania
ta
Tamil
ta_IN
Tamil
te
Telugu
te_IN
Telugu
th
Thai
th_TH
Thai
Somalia
Albania
India
India
Thailand
Administrator's Guide
About Locales
Locale Name
ti
Tigrinya
ti_ER
Tigrinya
Eritrea
ti_ET
Tigrinya
Ethiopia
tr
Turkish
tr_TR
Turkish
uk
Ukrainian
uk_UA
Ukrainian
ur
Urdu
ur_IN
Urdu
India
ur_PK
Urdu
Pakistan
uz
Uzbek
uz_Arab
Uzbek
Arabic
uz_Arab_AF
Uzbek
Afghanistan (Arabic)
uz_Cryl
Uzbek
Cyrillic
uz_Cryl_UZ
Uzbek
Uzbekistan (Cyrillic)
uz_Latin
Uzbek
Latin
us_Latin_UZ
Turkey
Ukraine
Uzbekistan (Latin)
vi
Vietnamese
vi_VN
Vietnamese
zh
Chinese
zh_Hans
Chinese
Simplified Han
zh_Hans_CN
Chinese
zh_Hans_HK
Chinese
zh_Hans_MO
Chinese
Vietnam
Administrator's Guide
About Locales
Locale Name
zh_Hans_SG
Chinese
zh_Hant
Chinese
Traditional Han
zh_Hant_HK
Chinese
zh_Hant_MO
Chinese
zh_Hant_TW
Chinese
zu
Zulu
zu_ZA
Zulu
South Africa
The locale setting is session scoped and applies only to queries (no DML/DDL) executed in that
session. You cannot specify a locale for an individual query.
You can set the default locale for new sessions using the DefaultSessionLocale configuration
parameter
Query related:
The following restrictions apply when queries are run with locale other than the default en_
US@collation=binary:
l
When one or more of the left-side NOT IN columns is CHAR or VARCHAR, multicolumn NOT
IN subqueries are not supported . For example:
=> CREATE TABLE test (x VARCHAR(10), y INT);
=> SELECT ... FROM test WHERE (x,y) NOT IN (SELECT ...);
ERROR: Multi-expression NOT IN subquery is not supported because a left hand
expression could be NULL
Note: Even if columns test.x and test.y have a NOT NULL constraint, an error occurs.
Administrator's Guide
About Locales
HAVING clause subqueries are not supported. In the following example, the GROUP BY x in the
outer query causes the error:
=> DROP TABLE test CASCADE;
=> CREATE TABLE test (x VARCHAR(10));
=> SELECT COUNT(*) FROM test t GROUP BY x HAVING x
IN (SELECT x FROM test WHERE t.x||'a' = test.x||'a' );
ERROR: subquery uses ungrouped column "t.x" from outer query
Subqueries that use analytic functions in the HAVING clause are not supported. For example:
=> DROP TABLE test CASCADE;
=> CREATE TABLE test (x VARCHAR(10));
=> SELECT MAX(x)OVER(PARTITION BY 1 ORDER BY 1)
FROM test GROUP BY x HAVING x IN (SELECT MAX(x) FROM test);
ERROR: Analytics query with having clause expression that involves aggregates and
subquery
is not supported
DML/DDL related:
l
SQL identifiers (such as table names and column names) can use UTF-8 Unicode characters.
For example, the following CREATE TABLE statement uses the (German eszett) in the table
name:
=> CREATE TABLE strae(x int, y int);
CREATE TABLE
Projection sort orders are made according to the default en_US@collation=binary collation.
Thus, regardless of the session setting, issuing the following command creates a projection
sorted by col1 according to the binary collation:
=> CREATE PROJECTION p1 AS SELECT * FROM table1 ORDER BY col1;
In such cases, strae and strasse are not stored near each other on disk.
Sorting by binary collation also means that sort optimizations do not work in locales other than
binary. HP Vertica returns the following warning if you create tables or projections in a nonbinary locale:
WARNING: Projections are always created and persisted in the default HP Vertica
locale. The current locale is de_DE
Administrator's Guide
About Locales
When creating pre-join projections, the projection definition query does not respect the locale or
collation setting. When you insert data into the fact table of a pre-join projection, referential
integrity checks are not locale or collation aware.
For example:
\locale LDE_S1
-- German
=> CREATE TABLE dim (col1 varchar(20) primary key);
=> CREATE TABLE fact (col1 varchar(20) references dim(col1));
=> CREATE PROJECTION pj AS SELECT * FROM fact JOIN dim
ON fact.col1 = dim.col1 UNSEGMENTED ALL NODES;
=> INSERT INTO dim VALUES('');
=> COMMIT;
The following INSERT statement fails with a "nonexistent FK" error even though '' is in the dim
table, and in the German locale 'SS' and '' refer to the same character.
=> INSERT INTO fact VALUES('SS');
ERROR: Nonexistent foreign key value detected in FK-PK join (fact x dim)
using subquery and dim_node0001; value SS
=> => ROLLBACK;
=> DROP TABLE dim, fact CASCADE;
When the locale is non-binary, HP Vertica uses the COLLATION function to transform the input
to a binary string that sorts in the proper order.
This transformation increases the number of bytes required for the input according to this
formula:
result_column_width = input_octet_width * CollationExpansion + 4
CHAR fields are displayed as fixed length, including any trailing spaces. When CHAR fields are
processed internally, they are first stripped of trailing spaces. For VARCHAR fields, trailing
spaces are usually treated as significant characters; however, trailing spaces are ignored when
sorting or comparing either type of character string field using a non-BINARY locale.
Administrator's Guide
Appendix: Binary File Formats
Note: You cannot mix Binary and ASCII source files in the same COPY statement.
File Signature
The first part of a NATIVE binary file consists of a file signature. The contents of the signature are
fixed, and listed in the following table.
Byte Offset
10
Hex Value
4E
41
54
49
56
45
0A
FF
0D
0A
00
E'\n'
E'\317'
E'\r'
E'\n'
E'\000'
Text Literals N
The purpose of the required signature is to ensure that the file has neither been corrupted by a non8-bit file transfer, nor stripped of carriage returns, linefeeds, or null values. If the signature is intact,
HP Vertica determines that the file has not been corrupted.
Column Definitions
Following the file signature, the file must define the widths of each column in the file as follows.
Administrator's Guide
Appendix: Binary File Formats
Byte
Length
Offset (bytes)
Description Comments
11
Header area
length
is the number of bytes from the end of this value to the start
of the row data.
15
NATIVE file
version
17
Filler
Always 0.
18
Number of
columns
4 bytes for
Column
each
widths
20+
column of
data in the
table
Note: All integers in NATIVE files are in little-endian format (least significant byte first).
The width of each column is determined by the data type it contains. The following table explains
the column width needed for each data type, along with the data encoding.
Data Type
Length
(bytes)
Column Content
INTEGER
1, 2, 4, 8
8-, 16-, 32-, and 64-bit integers are supported. All multi-byte
values are stored in little-endian format.
Note: All values for a column must be the width you specify here.
If you set the length of an INTEGER column to be 4 bytes, then
all of the values you supply for that column must be 32-bit
integers.
BOOLEAN
Administrator's Guide
Appendix: Binary File Formats
Data Type
Length
(bytes)
Column Content
FLOAT
CHAR
User-
specified
VARCHAR
4-byte
integer
(length) +
data
DATE
TIME
Administrator's Guide
Appendix: Binary File Formats
Data Type
Length
(bytes)
Column Content
TIMETZ
TIMESTAMPTZ 8
INTERVAL
BINARY
User-
specified
header in the Field Lengths entry for the field. The field in the
record must contain length number of bytes. If the value is
smaller than the specified length, the remainder should be filled
with nulls (E'\000').
VARBINARY
4-byte
integer +
data
Administrator's Guide
Appendix: Binary File Formats
Data Type
Length
(bytes)
NUMERIC
Column Content
scale)
(precision
19 + 1)
8 rounded
up
Row Data
Following the file header is a sequence of records that contain the data for each row of data. Each
record starts with a header:
Length (bytes)
Description Comments
Row length
Number of
Null value
columns 8
bit field
rounded up
(CEILING
(NumFields /
(sizeof
(uint8) * 8)
); )
the next most significant bit indicates whether the next column
Administrator's Guide
Appendix: Binary File Formats
Following the record header is the column values for the row. There is no separator characters for
these values. Their location in the row of data is calculated based on where the previous column's
data ended. Most data types have a fixed width, so their location is easy to determine. Variablewidth values (such as VARCHAR and VARBINARY) start with a count of the number of bytes the
value contains.
See the table in the previous section for details on how each data type's value is stored in the row's
data.
Example
The example below demonstrates creating a table and loading a NATIVE file that contains a single
row of data. The table contains all possible data types.
=> CREATE TABLE allTypes (INTCOL INTEGER,
FLOATCOL FLOAT,
CHARCOL CHAR(10),
VARCHARCOL VARCHAR,
BOOLCOL BOOLEAN,
DATECOL DATE,
TIMESTAMPCOL TIMESTAMP,
TIMESTAMPTZCOL TIMESTAMPTZ,
TIMECOL TIME,
TIMETZCOL TIMETZ,
VARBINCOL VARBINARY,
BINCOL BINARY,
NUMCOL NUMERIC(38,0),
INTERVALCOL INTERVAL
);
=> COPY allTypes FROM '/home/dbadmin/allTypes.bin' NATIVE DIRECT;
=> \pset expanded
Expanded display is on.
=> SELECT * from allTypes;
-[RECORD 1 ]--+-----------------------INTCOL
| 1
FLOATCOL
| -1.11
CHARCOL
| one
VARCHARCOL
| ONE
BOOLCOL
| t
DATECOL
| 1999-01-08
TIMESTAMPCOL
| 1999-02-23 03:11:52.35
TIMESTAMPTZCOL | 1999-01-08 07:04:37-05
TIMECOL
| 07:09:23
TIMETZCOL
| 15:12:34-04
VARBINCOL
| \253\315
BINCOL
| \253\315
NUMCOL
| 1234532
INTERVALCOL
| 03:03:03
The content of the allTypes.bin file appears below as a raw hex dump:
4E 41 54 49 56 45 0A FF 0D 0A 00 3D 00 00 00 01 00 00 0E 00
Administrator's Guide
Appendix: Binary File Formats
08
08
FF
00
65
FF
C0
AB
00
00
00
FF
00
20
FF
2E
CD
64
00
00
FF
01
20
FF
98
AB
D6
00
00
FF
00
20
FF
FF
CD
12
08
08
03
00
20
30
05
00
00
00
00
00
00
20
85
00
00
00
00
00
00
00
20
B3
00
00
00
00
00
00
00
20
4F
00
00
00
0A
08
18
00
03
7E
D0
00
00
00
00
00
00
00
E7
97
00
C0
00
00
00
C3
00
FF
01
00
47
00
00
00
F5
00
FF
80
00
A3
FF
08
08
28
4F
40
F0
00
8E
FF
00
00
5C
4E
1F
79
00
02
FF
00
00
8F
45
3E
F0
00
00
FF
00
00
C2
01
64
10
00
00
01
08
73
F1
9A
E8
02
00
00
00
00
00
BF
FE
E3
00
00
00
00
00
6F
FF
FF
00
00
00
00
00
6E
FF
FF
00
00
The following table breaks this file down into each of is components, and describes the values it
contains.
Hex Values
Description
Value
4E 41 54 49 56
45 0A FF 0D 0A
00
Signature
NATIVE\n\317\r\n\000
3D 00 00 00
Header area
61 bytes
length
01 00
Version 1
version
00
Filler value
0E 00
Number of
14 columns
columns
08 00 00 00
Width of column
8 bytes
1 (INTEGER)
08 00 00 00
Width of column
8 bytes
2 (FLOAT)
0A 00 00 00
Width of column
10 bytes
3 (CHAR(10))
FF FF FF FF
Width of column
4 (VARCHAR)
01 00 00 00
Width of column
1 bytes
5 (BOOLEAN)
08 00 00 00
Width of column
8 bytes
6 (DATE)
Administrator's Guide
Appendix: Binary File Formats
Hex Values
Description
Value
08 00 00 00
Width of column
8 bytes
7 (TIMESTAMP)
08 00 00 00
Width of column
8 bytes
8
(TIMESTAMPTZ)
08 00 00 00
Width of column
8 bytes
9 (TIME)
08 00 00 00
Width of column
8 bytes
10 (TIMETZ)
FF FF FF FF
Width of column
11
(VARBINARY)
03 00 00 00
Width of column
3 bytes
12 (BINARY)
18 00 00 00
Width of column
13 (NUMERIC)
08 00 00 00
Width of column
8 bytes
14 (INTERVAL).
last portion of the
header section.
73 00 00 00
Number of bytes
115 bytes
null values
contained in the
first row of data
Administrator's Guide
Appendix: Binary File Formats
Hex Values
Description
Value
01 00 00 00 00
00 00 00
INTEGER
column
C3 F5 28 5C 8F
C2 F1 BF
6F 6E 65 20 20
20 20 20 20 20
CHAR(10)
-1.11
FLOAT column
column
03 00 00 00
The number of
3 bytes
bytes in the
following
VARCHAR
value.
4F 4E 45
"ONE"
VARCHAR
column
01
True
BOOLEAN
column
9A FE FF FF FF
FF FF FF
30 85 B3 4F 7E
E7 FF FF
1999-01-08
DATE column
1999-02-23 03:11:52.35
TIMESTAMP
column
40 1F 3E 64 E8
E3 FF FF
1999-01-08 07:04:37-05
TIMESTAMPTZ
column
C0 2E 98 FF 05
00 00 00
D0 97 01 80 F0
79 F0 10
07:09:23
TIME column
15:12:34-05
TIMETZ column
Administrator's Guide
Appendix: Binary File Formats
Hex Values
Description
Value
02 00 00 00
The number of
2 bytes
bytes in the
following
VARBINARY
value
AB CD
VARBINARY
column
AB CD
BINARY column
00
00
00
00
00
00
00
00
00
00
00
00
00
64
00
00
00
00
D6
00
00
00
00
12
00
C0 47 A3 8E 02
00 00 00
1234532
NUMERIC
column
03:03:03
INTERVAL
column
See Also
l
COPY