Page Comparison

...

{scrollbar}

The steps in this guide include:

1. Installing a plugin from the Marketplace
2. Ingesting a Mainframe file into HDFS
3. Developing a PDI Transformation as a Mapper Task
4. Developing a PDI Job to Orchestrate all of the components
5. Executing and Reviewing Output

Prerequisites

In order to follow along with this how-to guide you will need the following:

• Hadoop
• Pentaho Data Integration
• Pentaho configured properly for your Distribution Configuring Pentaho for your Hadoop Distro and Version (Pentaho Suite Version 5.1)
• Complete the Using Pentaho MapReduce to Parse Weblog Data example Using Pentaho MapReduce to Parse Weblog Data
• LegStar z/OS File reader Plugin installed through the Marketplace (see below)

...

1. To install the plugin, launch Sppon and go to the menu Help -> Marketplace

2. Search for z/os and you should see the LegStar z/OS File reader Plugin

...

3. Click the Install this Plugin button

4. You need to add tools.jar to your PDI classpath, a simple way to do this is copy tools.jar from JDK_HOME/lib to data-integration/lib

5. Restart Spoon

Sample Files

...

Start Hadoop if not already running

Unzip the Data.zip to a directory

Create a Transformation to Convert z/OS File

1. Start PDI on your desktop. Choose File -> New -> Transformation from the menu.

2. Add a z/OS File Input step (if you don't see this step check the pre-requisites for how to install through the Marketplace). Your transformation should look like this:

...

3. Edit the z/OS File Input step. For z/OS filename you will select your Mainframe file. If using the sample data, this is under mf-files/ZOS.FCUSTDAT.bin. This file contains messages of a variable length, so check that box as well. This uses the IBM01140 character set. You will need to know what codepage your EBCDIC file uses: {+}http://en.wikipedia.org/wiki/EBCDIC+ 

4. Configure the COBOL copybook. Go to the COBOL tab to place your copybook. The copybook is used to "translate" the Mainframe file into fields. If you are using the sample files, you can select Import COBOL and browse to copybooks/CUSTDATCC. Note that in the file browser you must change the drop-down to All files so this will show up in the list. You should now see your copybook has been loaded.

5. Configure the Fields. Click the Get Fields button so that PDI can parse the copybook definition and determine what fields will be present.

The fields should be properly read in like this:

6. Preview Data. Hit the Preview button to make sure that all of the settings are correct.

...

Create a Mapper Transformation to Process Converted File in Hadoop

1. Choose File -> New -> Transformation from the menu.

...

5. Configure Split Fields. Double click the Split Fields step to configure the new fields that will be created. These are the same fields that were defined in the zos-to-hdfs transformation in part 1. Make sure that the field to split is "value" and that the delimiter matches what you configured in the zos-to-hdfs transformation.

...

6. Normalize the Transactions. You may have noticed that there are 5 transactions in each row. The way this COBOL copybook works, there can be anywhere from 0 to 5 transactions in each record. In a modern database, you would usually put each transaction in its own row. We can use the Row Normaliser step to pivot those individual transactions each into their own row. Add a Row Normaliser step and create a hop from Split Fields to Row Normaliser.

...

13. Create Value. Double click to edit the Concat Fields step. For the step name, change to "Create value", make the Target Field Name "newvalue", and make the Seperator a pipe "|". Then click the Get Fields button and you will see all fields in your stream are added. Remove key and newkey, and you should be left with 9 fields total.
To remove extra spaces that are from the original Mainframe file, make the following adjustments:

• CustomerId: Set Format to #.#
• CustomerName: Set Trim Type to "both"
• CustomerAddress: Set Trim Type to "both"
• TransactionNbr: Set Format to #.#
• Tx Amount: Set Format to #.#

14. Add MapReduce Output. Now that we have our new key and new value, we're ready for these to be returned by our mapper. Add the MapReduce Output step, and create a hop from Create value to MapReduce Output.

...

Create a Job to orchestrate the z/OS to HDFS and Pentaho MapReduce

1. Within PDI, choose File -> New -> Job from the menu system.

...