Versions Compared

Old Version 30

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
Include Page
Warning - Pentaho 5.2, 5.3, and 5.4 - Configuration
Warning - Pentaho 5.2, 5.3, and 5.4 - Configuration

Additional Configuration for YARN Shims

To get information on configuring EMR shims, click here.

Configuring CDH, HDP, and MapR Shims

Prerequisite: Make sure that your client has been configured so it can access any node in your cluster.

  1. Set the active Hadoop distribution.
  2. Configure the cluster settings.  Make sure that you

NOTE: For all supported hadoop distributions except MapR, add the yarn user on the cluster to the group defined by dfs.permissions.superusergroup property.

...

The dfs.permissions.superusergroup property can be found in

...

hdfs-site.xml file on your cluster or in the cluster management application.
3. Navigate to the folder that contains the shim, then open the yarn-site.xml file in a text editor.  Adjust the following parameters as needed.

Parameter

Values

yarn.application.classpath

  • HDP 2.2 - 

    Code Block
    <property> <name>yarn.application.classpath</name>
 <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,
/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,
/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,
/usr/hdp/current/hadoop-yarn-client/lib/*</value>
 </property>
  • MapR 4.0.1 Windows Client - 

    Code Block
    <property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*
:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*
:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*
:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:/usr/share/aws/emr/emrfs/lib/*
:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$PWD/*:%PWD%/*
</value>
</property>
  • All other shims - Classpaths needed to execute YARN applications.  Separate paths with a comma.

yarn.resourcemanger.hostname

  • CDH 5.x - Update the hostname in your environment or use the default: clouderamanager.cdh5.test
  • HDP 2.x - Update the hostname in your environment or use the default:  sandbox.hortonworks.com
  • All other shims: Hostname in your environment.

yarn.resourcemanager.address

  • All shims: Update hostname and port to match your environment.

yarn.resourcemanager.admin.address

  • All shims: Update hostname and port to match your environment.

...

4. (CDH 5.1 and 5.2 only): * Navigate to the folder that contains the shim, then open the hive-site.xml file in a text editor. Modify the hive.metastore.uris property so that it points to the location of your hive metastore. Save and close the hive-site.xml file.

5. (All shims): Navigate to the folder that contains the shim, then open the mapred-site.xml file, in a text editor. Then make the changes in the table.

Parameter

Value

mapreduce.jobhistory.address

Set this to the place where job history logs are stored.

mapreduce.app-submission.cross-platform

Add this parameter to the mapred-site.xml file between the <property> tags, then set it to true, like this:


Code Block
<name>mapreduce.app-submission.cross-platform</name>
<value>true</value>



This property allows mapreduce jobs to run on windows and linux platforms, and vice versa.

6. (HDP 2.2): In the mapred-site.xml file, make the changes shown in the table.

Parameter

Value

mapreduce.application.classpath 

Add this parameter to the mapred-site.xml file between the <property> tags.

Code Block
<property>
    <name>mapreduce.application.classpath</name>
    <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*
:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*
:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*
:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*
:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*
:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure
</value>
  </property>


mapreduce.application.framework.path

Add this parameter to the mapred-site.xml file between the <property> tags.

Code Block
<property>
 <name>mapreduce.application.framework.path</name>
 <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
 </property>


...

Code Block
java.system.hdp.version=2.2.0.0-2041

Note: Not all shim properties can be set in the Spoon user interface, nor are instructions for modifying them listed here. If you need to set additional properties that are not addressed in these instructions, you will need to set them manually in the *-site.xml files that are in the shim directory. Consult your Hadoop distribution's vendor for details about the properties you want to set.

High Availability for CDH 5.3

Note: If you are configuring CDH 5.3 to be used in High Availability mode, we recommend that you use the Cloudera Manager "Download Client Configuration" feature. The Download Client Configuration feature provides a convenient way to get configuration files from the cluster for a service (such as HBase, HDFS, or Yarn). Use this feature to download the unzip the configuration zip files to the pentaho-big-data-plugin/hadoop-configurations/cdh5x directory.​For more information on how to do this, see Cloudera documentation:
http://www.cloudera.com/content/cloudera/en/documentation/core/v5-3-x/topics/cm_mc_client_config.html.

Anchor
emr
emr

Configuring EMR Shims

...

Note:  Pentaho does not support HBase on HDFS in EMR 3.4.

Prerequisites

...