Description

This step writes data to an HBase table according to user-defined column metadata.

Options

Configure Connection Tab

This tab contains HBase connection information. You can configure a connection in one of two ways: using the Hadoop cluster properties, or by using an hbase-site.xml and (an optional) hbase-default.xml configuration file.

Below the connection details are fields to specify the target HBase table to write to and a mapping by which to encode incoming field values. Before a value can be written to HBase it is necessary to tell the step which column family it belongs to and what its type is. Furthermore, it is necessary to specify type information about the key of the table. This is where a mapping comes into play and the user must define a mapping to use for a new or given target table.

Hadoop Cluster

The Hadoop cluster configuration dialog allows you to specify configuration detail such as host names and ports for HDFS, Job Tracker, and other big data cluster components, which can be reused in transformation steps and job entries that support this feature.

Create and Edit Mappings Tab

This tab creates or edits a mapping for a given HBase table. A mapping simply defines metadata about the values that are stored in the table. Since just about all information is stored as raw bytes in HBase, this allows PDI to decode values and execute meaningful comparisons for column-based result set filtering. Names of fields entering the step are expected to match the aliases of fields defined in the mapping. All incoming fields must have a matching counterpart in the mapping. There may be fewer incoming fields than defined in the mapping, but if there are more incoming fields then an error will occur. One of the incoming fields must match the key defined in the mapping.

This tab operates in the same way as it does in the HBase Input step, with the exception that it allows the target HBase table to be created if it doesn't already exist. Furthermore, the fields coming into the step can be used as the basis for defining a mapping.

Selecting a table will populate the Mapping name drop-down box with the names of any mappings that exist for the table. This box will be empty if there are no mappings defined for the selected table and the user can enter the name of a new mapping. The fields area of the tab can be used to enter information about the columns in the HBase table that the user wants to map. Selecting the name of an existing mapping will load the fields defined in that mapping into the fields area of the display. Alternatively, if the user can create a new HBase table and mapping for it simultaneously by configuring the fields of the mapping and entering the name of a table that doesn't exist in the Hbase table name drop down box.

A valid mapping must define meta data for at least the key of the source HBase table. The key must have an Alias specified because there is no name given to the key of an HBase table. Non-key columns must specify the Column family that they belong to and the Column name. An*Alias* is optional - if not supplied then the column name is used. All fields must have type information supplied.

To enable meaningful range scans over key values it is necessary for keys to sort properly in HBase. This is why we make a distinction between unsigned and signed numbers. Since HBase stores integers/longs in twos complement internally it is necessary to flip the sign bit before storing a signed number in order to have positive numbers sort after negative ones. Unsigned integers/longs are assumed to have been stored directly without without inverting the sign bit. Note that keys that are dates are assumed to have been stored as just signed or unsigned longs (i.e. number of milliseconds elapsed since the epoch). If you have a key that is date in string format (ISO 8601 date-time format is designed such that lexicographic order corresponds to chronological order) it can be mapped as type String and then PDI can be used to change the type to Date for manipulation in the transformation.

No distinction is made between signed and unsigned numbers here because no sorting is performed by HBase on anything other than the key. Boolean values may be stored in HBase as 0/1 integer/long or as strings (Y/N, yes/no, true/false, T/F). BigNumber may be stored as either a serialized BigDecimal object or in string form (i.e. a string that can be parsed by BigDecimal's constructor). Serializable is any serialized Java object and Binary is a raw array of bytes. String columns may optionally have a set of legal values defined for them by entering comma-separated data into the Indexed values column in the fields table.

Pressing the Save mapping button will save the mapping. If the target table does not exist, then the user will be asked if they want to use key and column family information from the mapping as the basis for the structure of the new table. The user will also be alerted to any missing information in the mapping definition and prompted to rectify it before the mapping is actually saved. Note that it is possible to define multiple mappings - involving different subsets of columns - on the same HBase table. The Delete mapping button allows the current named mapping for the current named table to be deleted from the mapping table. Note that this does not delete the actual HBase table in question.

To speed up the creation of a mapping it is possible to use the incoming fields to the step as the basis for the mapping. Pressing the Get incoming fields button will populate the mapping table with information from the fields entering the step. The Alias and Column name of each mapping field will be set to the name of an incoming field, the type information will be filled in automatically, and the Column family will be set to either the name of the first column family defined if the table already exists, or, a default value ("Family1"), which can be altered by the user to define their own families when the target table is created. Note that the step does not support adding new column families to an existing table.

Performance Considerations

HBase server configuration and tuning aside, there are several things to consider when using HBase Output. The Configure connection tab provides a field for setting the size of the write buffer used to transfer data to HBase. The user may enter a number here (in bytes). A larger buffer consumes more memory (on both the client and server sides) but results in fewer remote procedure calls. The default (in the hbase-default.xml) is 2Mb (2097152 bytes) and this value will be used if the field is left blank. When left blank, auto flush is enabled and "Put" operations are executed immediately. This means that each row will be transmitted to HBase at the time it arrives at the step. Entering a number (even if it is the same as the default) for the size of the write buffer will disable auto flush and will result in incoming rows only being transferred once the buffer is full.

There is also a check box for disabling writing to the Write Ahead Log (WAL). The WAL is used as a lifeline to restore the status quo if the server goes down while data is being inserted. However, the tradeoff for error-recovery is speed.

NOTE: The Create/edit mappings tab allows the user to specify some options for when a new table is created in HBase. In the HBase table name field the user may optionally suffix the name of the new table with parameters for specifying what compression to use and whether or not to use Bloom filters to speed up lookups. The options for compression include NONE, GZ and LZO; the options for Bloom filters include NONE, ROW, ROWCOL. If nothing is specified - beyond the name of the new table - by the user, then the default of NONE is used for both compression and Bloom filters. For example, the following string entered in the HBase table name field specifies that a new table called "NewTable" should be created with GZ compression and ROWCOL Bloom filters:

NewTable@GZ@ROWCOL

Note that due to licensing constraints, HBase does not ship with LZO compression libraries - these must be installed on each node by the user if they wish to use LZO compression.

Metadata Injection Support (7.x and later)

All fields of this step support metadata injection. You can use this step with ETL Metadata Injection to pass metadata to your transformation at runtime.