SQOOP Hcatalog Integration

SQOOP HCatalog Integration
Venkat Ranganathan
Sqoop Meetup
HCatalog Overview
Sqoop HCatalog integration Goals
HCatalog Overview
• Table and Storage Management Service for
– Enables PIG/MR and Hive to more easily share
data on the grid
• Uses the Hive Meta-store.
• Abstracts location and format of the data
• Supports reading and writing files in any
format for which there is a Hive Serde
• Now part of Hive.
Sqoop HCatalog Integration Goals
• Support HCatalog features consistent with
Sqoop usage.
– Support both imports into and exports from
HCatalog table
– Enable Sqoop read and write data in various
– Automatic table schema mapping
– Data fidelity
– Support for static and dynamic partition keys
Support imports and exports
• Allows the HCatalog tables to be either the
source or destination of a Sqoop job.
• In an HCatalog import, target-dir and
warehouse-dir are replaced with the
HCatalog table name.
• Similarly for exports, the export directory is
substituted with the HCatalog table name.
File format support
• HCatalog integration into Sqoop now
enables Sqoop to
– Import/Export files of various formats that
have hive serde created
– Textfiles, Sequence files, RCFiles, ORCFile,…
– This makes Sqoop agnostic of the file format
used which can change over time based on
new innovations/needs.
Automatic table schema mapping
• Sqoop allows a hive table to be created
based on the enterprise data store schema
• This is enabled for HCatalog table imports as
• Automatic mapping with optional user
• Ability to provide a storage options for the
newly created table.
• All HCatalog primitive types supported
Data fidelity
• With Text based imports (as in Sqoop
hive-import option), the text values have to
be massaged so that delimiters are not
• Sqoop provides two options to handle this.
• Error prone and data is modified to be
stored on Hive
Data fidelity
• With HCatalog table imports to file formats
like RCFile, ORCFile etc, there is no need
to strip these delimiters in column values.
• Data is preserved without any massaging
• If the target Hcatalog table file format is
Text, then the two options can still be used
as before.
Support for static and dynamic partitioning
• HCatalog tables partition keys can be
dynamic or static.
• Static partitioning keys have values
provided as part of the DML (known at
Query compile time)
• Dynamic partitioning keys have values
provided at execution time.
– Based on value of a column being imported
Support for static and dynamic partitioning
• Both types of tables supported during
• Multiple partition keys per table are
• Only one can be a static partition key can
be specified (Sqoop restriction).
• Only table with one partitioning key can be
automatically created.
• Future proof your Sqoop jobs by making
them agnostic of file-formats used
• Remove additional steps before taking
data to the target table format
• Preserve data contents
Availability & Documentation
• Part of Sqoop 1.4.4 release
• A chapter devoted to HCatalog integration
in the User Guide
• URL:
© Hortonworks Inc. 2013
© Hortonworks Inc. 2013

similar documents