Create Mapper and Reducer for Aggregate Dataset

Owned by Former user (Deleted)
Last updated: Jun 27, 2012 by Former user (Deleted)Version comment
7 min read

Create a Pentaho Mapper Transformation

In this task you will create a Pentaho Mapper transformation. This transformation will be used to run a Pentaho MapReduce job on the Hadoop cluster. This transformation will consume a parsed, tab-delimited weblog record and construct intermediate data consisting of a key comprised of Client IP, Year, and Month and emitting a constant value of 1. The value denotes a single pageview for the key. The summing will be done by the Reducer – which we will develop next.

  1. Start PDI on your desktop. Once it is running choose 'File' -> 'New' -> 'Transformation' from the menu system or click on the 'New file' icon on the toolbar and choose the 'Transformation' option.

    Speed Tip

    You can download the Kettle Transform aggregate_mapper.ktr already completed

  2. Add a MapReduce Input Step: You are going to read data into the transformation from MapReduce, so expand the 'Big Data' section of the Design palette and drag a 'MapReduce Input' node onto the transformation canvas. Your transformation should look like:


  3. Edit the MapReduce Input Step: Double-click on the 'MapReduce Input' node to edit its properties. Enter this information:
    1. Key Field Type: Enter String
    2. Value Field Type: Enter String
      When you are done your 'MapReduce Input' window should look like this:

      Click 'OK' to close the window.

  4. Add a Split Fields Step: You need to split the incoming records on tab to get the individual fields in the record, so expand the 'Transform' section of the Design palette and drag a 'Split Fields' node onto the transformation canvas. Your transformation should look like:


  5. Connect the Input and Split Fields Steps: Hover the mouse over the 'MapReduce Input' node and a tooltip will appear.
    Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Split Fields' node. Your canvas should look like this:


  6. Edit the Split Fields Step: Double-click on the 'Split Fields' node to edit its properties. Enter this information:
    1. Field to split: Select 'value'
    2. Delimiter: Enter '$[09]'  09 is the hexadecimal representation of the ASCII tab character.
    3. Fields: The field list will be the following of with 'Type' set to 'String'
      1. client_ip
      2. full_request_date
      3. day
      4. month
      5. month_num
      6. year
      7. hour
      8. minute
      9. second
      10. timezone
      11. http_verb
      12. uri
      13. http_status_code
      14. bytes_returned
      15. referrer
      16. user_agent
        When you are done your 'MapReduce Input' window should look like this:

        Click 'OK' to close the window.

  7. Add a User Defined Java Expression Step: You need to concatenate the client_ip, year, and month together to create the key field, so expand the 'Scripting' section of the Design palette and drag a 'User Defined Java Expression' node onto the transformation canvas. Your transformation should look like:


  8. Connect the Split Fields and User Defined Java Expression Steps: Hover the mouse over the 'Split Fields' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'User Defined Java Expression' node. Your canvas should look like this:


  9. Edit the User Defined Java Expression Step: Double-click on the 'User Defined Java Expression' node to edit its properties. Do the following:
    1. Create a new field 'new_key' with Type 'String' and the following Java expression: 
      client_ip + '\t' + year + '\t' + month_num
      Note the characters between the '' are tabs. You will have to copy and paste tab characters into the Java Expression.
    2. Create a new field 'new_value' with Type 'Integer' and the Java expression '1'.
      When you are done your window should look like:

      Click 'OK' to close the window.

  10. Add a MapReduce Output Step: You need to write the new key and new value to the output, so expand the 'Big Data' section of the Design palette and drag a 'MapReduce Output' node onto the transformation canvas. Your transformation should look like:


  11. Connect the Java Expression and Output Steps: Hover the mouse over the 'User Defined Java Expression' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'MapReduce Output' node. Your canvas should look like this:


  12. Edit the Output Step: Double-click on the 'MapReduce Output' node to edit its properties. Enter the following information:
    1. Key field: Select 'new_key'
    2. Value field: Select 'new_value'
      When you are done your window should look like:

      Click 'OK' to close the window.

  13. Save the Transformation: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'aggregate_mapper.ktr' into a folder of your choice.

Create a Pentaho Reducer Transformation

In this task you will create a Pentaho reducer transformation. It will consume the output of the mapper you just created and emit the same key while summing the values. The emitted value will be the count of pageviews for the key. Note that we do not need to do any sorting in this transformation since we can rely on the MapReduce engine's sort and shuffle process to ensure that the reducer receive records sorted in key order.

  1. Start PDI on your desktop. Once it is running choose 'File' -> 'New' -> 'Transformation' from the menu system or click on the 'New file' icon on the toolbar and choose the 'Transformation' option.

    Speed Tip

    You can download the Kettle Transform aggregate_reducer.ktr already completed

  2. Add a MapReduce Input Step: You are going to read data into the transformation from MapReduce, so expand the 'Big Data' section of the Design palette and drag a 'MapReduce Input' node onto the transformation canvas. Your transformation should look like:


  3. Edit the MapReduce Input Step: Double-click on the 'MapReduce Input' node to edit its properties. Enter this information:
    1. Key Field Type: Enter 'String'
    2. Value Field Type: Enter 'Integer'
      When you are done your 'MapReduce Input' window should look like this:

      Click 'OK' to close the window.

  4. Add a Group By Step: You are going to sum the values for each key, so expand the 'Statistics' section of the Design palette and drag a 'Group by' node onto the transformation canvas. Your transformation should look like:


  5. Connect the Input and Group By Steps: Hover the mouse over the 'MapReduce Input' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Group by' node. Your canvas should look like this:


  6. Edit the Group By Step: Double-click on the 'Group by' node to edit its properties. Enter this information:
    1. Group field: Enter 'key'
    2. Aggregates: Enter the following:
      1. Name: Enter 'new_value'
      2. Subject: Enter 'value'
      3. Type: Select 'Sum'
        When you are done your window should look like this:

        Click 'OK' to close the window.

  7. Add a MapReduce Output Step: You need to write the new key and new value to the output, so expand the 'Big Data' section of the Design palette and drag a 'MapReduce Output' node onto the transformation canvas. Your transformation should look like:


  8. Connect the Group By and Output steps: Hover the mouse over the 'Group by' node and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'MapReduce Output' node. Your canvas should look like this:


  9. Edit the Output Step: Double-click on the 'MapReduce Output' node to edit its properties. Enter the following information:
    1. Key field: Select 'key'
    2. Value field: Select 'new_value'
      When you are done your window should look like:

      Click 'OK' to close the window.

  10. Save the Transformation: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'aggregate_reducer.ktr' into a folder of your choice.