| Wiki Markup |
|---|
{scrollbar}
{ |
| Excerpt |
|---|
...
How |
...
to |
...
use |
...
Pentaho |
...
MapReduce |
...
to |
...
convert |
...
raw |
...
weblog |
...
data |
...
into |
...
parsed, |
...
delimited |
...
records. |
...
The
...
steps
...
in
...
this
...
guide
...
include
...
- Loading
...
- the
...
- sample
...
- data
...
- file
...
- into
...
- CLDB
...
- Developing
...
- a
...
- PDI
...
- transformation
...
- which
...
- will
...
- serve
...
- as
...
- a
...
- Mapper
...
- Developing
...
- a
...
- PDI
...
- job
...
- which
...
- will
...
- invoke
...
- a
...
- Pentaho
...
- MapReduce
...
- step
...
- that
...
- runs
...
- a
...
- map-only
...
- job,
...
- using
...
- the
...
- developed
...
- mapper
...
- transformation.
...
- Executing
...
- and
...
- reviewing
...
- output
...
Prerequisites
In order to follow along with this how-to
...
guide
...
you
...
will
...
need
...
the
...
following:
...
- MapR
- Pentaho Data Integration
- Pentaho Hadoop Node Distribution
Sample Files
The sample data file needed for this guide is:
File Name | Content |
...
Unparsed, |
...
raw |
...
weblog |
...
data |
NOTE:
...
If
...
you
...
have
...
completed
...
the
...
...
...
...
...
...
...
guide,
...
then
...
the
...
necessary
...
file
...
will
...
already
...
be
...
in
...
the
...
proper
...
location.
...
This
...
file
...
should
...
be
...
placed
...
in
...
CLDB
...
at
...
/weblogs/raw
...
using
...
the
...
following
...
commands.
...
| Code Block |
|---|
hadoop fs -mkdir /weblogs
hadoop fs -mkdir /weblogs/raw
hadoop fs -put weblogs_rebuild.txt /weblogs/raw/{code}
h1. |
Step-By-Step
...
Instructions
Setup
Start MapR if it is not already running.
| Include Page | ||||
|---|---|---|---|---|
|
Create a PDI Job to Execute a Map Only MapReduce Process
In this task you will create a job that will execute a "map-only"
...
MapReduce
...
process
...
using
...
the
...
mapper
...
transformation
...
you
...
created
...
in
...
the
...
previous
...
section.
...
| Tip | ||||||||
|---|---|---|---|---|---|---|---|---|
| =
|
| }||||||
You can download the Kettle Job [_parse_mr.kjb|Using Pentaho MapReduce to Parse Weblog Data in MapR^weblog]already completed{tip} # Withincompleted |
- Within PDI,
...
- choose
...
- 'File'
...
- ->
...
- 'New'
...
- ->
...
- 'Job'
...
- from
...
- the
...
- menu
...
- system
...
- or
...
- click
...
- on
...
- the
...
- 'New
...
- file'
...
- icon
...
- on
...
- the
...
- toolbar
...
- and
...
- choose
...
- the
...
- 'Job'
...
- option.
...
- Add a Start Job Entry: You need to tell PDI where to start the job, so expand the 'General' section of the Design palette and drag a 'Start' node onto the job canvas. Your canvas should look like:
- Add a Pentaho MapReduce Job Entry: Expand the 'Big Data' section of the Design palette and drag a 'Pentaho MapReduce' job entry onto the job canvas. Your canvas should look like:
- Connect the Start and MapReduce Job Entries: Hover the mouse over the 'Delete folders' job entry and a tooltip will appear. Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Pentaho MapReduce' job entry.
Your canvas should look like this:
- Edit the MapReduce Job Entry: Double-click on the 'Pentaho MapReduce' job entry to edit its properties. Enter this information:
- Hadoop Job Name: Enter 'Web Log Parser'
- Mapper Transformation: Enter <PATH>/weblog
_parse_mapper.ktrAnchor _GoBack _GoBack
<PATH> is the folder path you saved the mapper in. - Mapper Input Step Name: Enter 'Map/Reduce Input'
- Mapper Output Step Name: Enter 'Map/Reduce Output'
When you are done the window should look like:
- Configure the MapReduce Job: Switch to the 'Job Setup' tab. Enter this information:
- Check 'Suppress Output of Map Key'
- Input Path: Enter '/weblogs/raw'
- Output Path: Enter '/weblogs/parse'
- Input Format: Enter 'org.apache.hadoop.mapred.TextInputFormat'
...
- Output
...
- Format:
...
- Enter
...
- 'org.apache.hadoop.mapred.TextOutputFormat'
...
- Check
...
- 'Clean
...
- output
...
- path
...
- before
...
- execution'
...
When
...
- you
...
- are
...
- done
...
- your
...
- window
...
- should
...
- look
...
- like:
- like:
- Configure the Cluster Properties: Switch to the 'Cluster' tab. Enter this information:
- Hadoop distribution: Select 'MapR'
- Working Directory: Enter '/tmp'
- HDFS Hostname, HDFS Port, Job Tracker Hostname, Job Tracker Port: Your connection information. For a local single node cluster leave blank.
- Number of Mapper Tasks: Enter '3'. You can play around with this to get the best performance based on the size of your data and the number of nodes in your cluster.
- Number of Reducer Tasks: Enter '0'
- Check 'Enable Blocking'
- Logging Interval: Enter '10'. The number of seconds between pinging MapR for completion status messages
When you are done your window should look like:
Click 'OK' to close the window.
- Save the Job: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'weblogs_parse_mr.kjb'
...
- into
...
- a
...
- folder
...
- of
...
- your
...
- choice.
...
- Run the Job: Choose 'Action'
...
- ->
...
- 'Run'
...
- from
...
- the
...
- menu
...
- system
...
- or
...
- click
...
- on
...
- the
...
- green
...
- run
...
- button
...
- on
...
- the
...
- job
...
- toolbar.
...
- An
...
- 'Execute
...
- a
...
- job'
...
- window
...
- will
...
- open.
...
- Click
...
- on
...
- the
...
- 'Launch'
...
- button.
...
- An
...
- 'Execution
...
- Results'
...
- panel
...
- will
...
- open
...
- at
...
- the
...
- bottom
...
- of
...
- the
...
- PDI
...
- window
...
- and
...
- it
...
- will
...
- show
...
- you
...
- the
...
- progress
...
- of
...
- the
...
- job
...
- as
...
- it
...
- runs.
...
- After
...
- a
...
- few
...
- seconds
...
- the
...
- job
...
- should
...
- finish
...
- successfully:
If any errors occurred the job entry that failed will be highlighted in red and you can use the 'Logging' tab to view error messages.
Check MapR for Parsed Weblog Data
- If you have mounted your MapR CLDB onto your local machine you may verify the file loaded by navigating to the MapR directory.
Code Block ls /mapr/my.cluster.com/weblogs/parse
...
This
...
- should
...
- return:
...
_logs
...
- part-00000
...
- part-00001
...
- part-00002
...
- _SUCCESS
...
- If you have not mounted your MapR CLDB onto your local machine you may alternatively check MapR by:
Code Block hadoop fs \-ls /weblogs/parse
...
This should return:
-rwxrwxrwx 3 demo demo 27132365 2012-01-04
...
- 16:52
...
- /weblogs/parse/part-00001
...
-rwxrwxrwx
...
- 3
...
- demo
...
- demo
...
- 0
...
- 2012-01-04
...
- 16:52
...
- /weblogs/parse/_SUCCESS
...
-rwxrwxrwx
...
- 3
...
- demo
...
- demo
...
- 27188268
...
- 2012-01-04
...
- 16:52
...
- /weblogs/parse/part-00002
...
drwxrwxrwx
...
- -
...
- demo
...
- demo
...
- 1
...
- 2012-01-04
...
- 16:52
...
- /weblogs/parse/_logs
...
-rwxrwxrwx
...
- 3
...
- demo
...
- demo
...
- 27147417
...
- 2012-01-04
...
- 16:52
...
- /weblogs/parse/part-00000
...
Summary
During this guide you learned how to create and execute a Pentaho MapReduce job to parse raw weblog data.
| Wiki Markup |
|---|
{scrollbar} |