Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Next »

Additional Configuration for YARN Shims

Prerequisite: Make sure that your client has been configured so it can access any node in your cluster.

  1. Set the active Hadoop distribution.
  2. Configure the cluster settings.
  3. Navigate to the folder that contains the shim, then open the yarn-site.xml file in a text editor.  Adjust the following parameters as needed.

Parameter

Values

yarn.application.classpath

  • HDP 2.2 -  
    <property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value>
  </property>
  • MapR 4.0.1 Windows Client -  

    <property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$PWD/*:%PWD%/*
</value>
</property>
  • All other shims - Classpaths needed to execute YARN applications.  Separate paths with a comma.

yarn.resourcemanger.hostname

  • CDH 5.x - Update the hostname in your environment or use the default: clouderamanager.cdh5.test
  • HDP 2.x - Update the hostname in your environment or use the default:  sandbox.hortonworks.com
  • All other shims: Hostname in your environment.

yarn.resourcemanager.address

  • All shims: Update hostname and port to match your environment.

yarn.resourcemanager.admin.address

  • All shims: Update hostname and port to match your environment.

There are a few more things that you need to do:

4. (CDH 5.1 and 5.2 only): * Navigate to the folder that contains the shim, then open the hive-site.xml file in a text editor. Modify the hive.metastore.uris property so that it points to the location of your hive metastore. Save and close the hive-site.xml file.

5. (All shims): Navigate to the folder that contains the shim, then open the mapred-site.xml file, in a text editor. Then make the changes in the table.

Parameter

Value

mapreduce.jobhistory.address

Set this to the place where job history logs are stored.

mapreduce.app-submission.cross-platform

Add this parameter to the mapred-site.xml file between the <property> tags, then set it to true, like this: 


<name>mapreduce.app-submission.cross-platform</name>
<value>true</value>



This property allows mapreduce jobs to run on windows and linux platforms, and vice versa.

6. (HDP 2.2): In the mapred-site.xml file, make the changes shown in the table.

Parameter

Value

mapreduce.application.classpath 

Add this parameter to the mapred-site.xml file between the <property> tags. 

<property>
    <name>mapreduce.application.classpath</name>
    <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
  </property>


mapreduce.application.framework.path

Add this parameter to the mapred-site.xml file between the <property> tags. 

<property>
 <name>mapreduce.application.framework.path</name>
 <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
 </property>


7. (HDP 2.2 only) In the HDP configuration.properties file on the client, add the following line.

java.system.hdp.version=2.2.0.0-2041

Note: Not all shim properties can be set in the Spoon user interface, nor are instructions for modifying them listed here. If you need to set additional properties that are not addressed in these instructions, you will need to set them manually in the *-site.xml files that are in the shim directory. Consult your Hadoop distribution's vendor for details about the properties you want to set.

  • No labels