...
1. To install the plugin, launch Sppon and go to the menu Help -> Marketplace
2. Search for z/os and you should see the LegStar z/OS File reader Plugin
...
3. Click the Install this Plugin button
4. You need to add tools.jar to your PDI classpath, a simple way to do this is copy tools.jar from JDK_HOME/lib to data-integration/lib
5. Restart Spoon
Sample Files
...
Start Hadoop if not already running
Unzip the Data.zip to a directory
Create a Transformation to Convert z/OS File
| Info |
|---|
This sample works by converting the existing Mainframe file that is in z/OS format into a text format for processing in Hadoop. |
1. Start PDI on your desktop. Choose File -> New -> Transformation from the menu.
2. Add a z/OS File Input step (if you don't see this step check the pre-requisites for how to install through the Marketplace). Your transformation should look like this:
...
4. Configure the COBOL copybook. Go to the COBOL tab to place your copybook. The copybook is used to "translate" the Mainframe file into fields. If you are using the sample files, you can select Import COBOL and browse to copybooks/CUSTDATCC. Note that in the file browser you must change the drop-down to All files so this will show up in the list. You should now see your copybook has been loaded.
5. Configure the Fields. Click the Get Fields button so that PDI can parse the copybook definition and determine what fields will be present.
| Note |
|---|
If you see an error that it was unable to find tools.jar, this is because tools.jar was not found in the classpath. A simple fix is to copy JDK_HOME/lib/tools.jar to data-integration/lib. |
The fields should be properly read in like this:
6. Preview Data. Hit the Preview button to make sure that all of the settings are correct.
...
Create a Mapper Transformation to Process Converted File in Hadoop
| Info |
|---|
In this task you will create a Mapper Transformation that will parse the Mainframe file into a structured format, and pivot each transaction into its own row. |
1. Choose File -> New -> Transformation from the menu.
...
5. Configure Split Fields. Double click the Split Fields step to configure the new fields that will be created. These are the same fields that were defined in the zos-to-hdfs transformation in part 1. Make sure that the field to split is "value" and that the delimiter matches what you configured in the zos-to-hdfs transformation.
| Tip |
|---|
...
You can copy a table from one transformation to another, but the columns must match up. If you look at the Hadoop File Output step from zos-to-hdfs, you will see that the columns are Name, Type, Format, Length, etc. However, the columns for the Split Fields step are New Field, ID, Remove ID, Type, Length, etc. You can copy from Hadoop File Output to Split Fields by using Excel to add the 2 new columns (ID and Remove ID). Start a new worksheet in Excel, then select all rows in Hadoop File Output, use CTRL-C to copy, and then CTRL-V to paste into Excel. Select Column B (Type), and hit Insert (do this twice). Now select all of your data in Excel, use CTRL-C to copy, and then CTRL-V to paste into the Split Fields Step.
|
6. Normalize the Transactions. You may have noticed that there are 5 transactions in each row. The way this COBOL copybook works, there can be anywhere from 0 to 5 transactions in each record. In a modern database, you would usually put each transaction in its own row. We can use the Row Normaliser step to pivot those individual transactions each into their own row. Add a Row Normaliser step and create a hop from Split Fields to Row Normaliser.
...
Create a Job to orchestrate the z/OS to HDFS and Pentaho MapReduce
| Info |
|---|
In this task you will create a job that will convert the Mainframe file to CSV within HDFS, then run a "map only" MapReduce process to create an output file. |
1. Within PDI, choose File -> New -> Job from the menu system.
...