...
Parameter | Values |
|---|
yarn.application.classpath | - HDP 2.2 -
| Code Block |
|---|
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value>
</property>
|
- MapR 4.0.1 Windows Client -
| Code Block |
|---|
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$PWD/*:%PWD%/*
</value>
</property>
|
- All other shims : - Classpaths needed to execute YARN applications. Separate paths with a comma.
|
yarn.resourcemanger.hostname | - CDH 5.x - Update the hostname in your environment or use the default: clouderamanager.cdh5.test
- HDP 2.x - Update the hostname in your environment or use the default: sandbox.hortonworks.com
- All other shims: Hostname in your environment.
|
yarn.resourcemanager.address | - All shims: Update hostname and port to match your environment.
|
yarn.resourcemanager.admin.address | - All shims: Update hostname and port to match your environment.
|
...
Parameter | Value |
|---|
mapreduce.jobhistory.address | Set this to the place where job history logs are stored. |
mapreduce.app-submission.cross-platform | Add this parameter to the mapred-site.xml file between the <property> tags, then set it to true, like this:
| Code Block |
|---|
<name>mapreduce.app-submission.cross-platform</name> |
This property allows mapreduce jobs to run on windows and linux platforms, and vice versa.
|
6. (HDP 2.2): In the mapred-site.xml file, make the changes shown in the table.
Parameter | Value |
|---|
mapreduce.application.classpath
| Add this parameter to the mapred-site.xml file between the <property> tags.
| Code Block |
|---|
<property>
<name>mapreduce.application.classpath</name>
<value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
</property>
|
|
mapreduce.application.framework.path
| Add this parameter to the mapred-site.xml file between the <property> tags.
| Code Block |
|---|
<property>
<name>mapreduce.application.framework.path</name>
<value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
</property> |
|
7. (HDP 2.2 only) In the HDP configuration.properties file on the client, add the following line.
| Code Block |
|---|
java.system.hdp.version=2.2.0.0-2041
|
Note: Not all shim properties can be set in the Spoon user interface, nor are instructions for modifying them listed here. If you need to set additional properties that are not addressed in these instructions, you will need to set them manually in the *-site.xml files that are in the shim directory. Consult your Hadoop distribution's vendor for details about the properties you want to set.