As part of this section we will see how to enable HDFS Namenode High Availability as well as YARN Resource Manager High Availability while exploring key concepts. High Availability – Overview Configure HDFS Namenode HA Review Properties – HDFS Namenode HA HDFS Namenode HA – Key Concepts Configure YARN Resource Manager HA Review – YARN …
Now let us talk about Map Reduce Job Execution Life Cycle. While YARN is Resource Management framework, Map Reduce is distributed data processing framework. On Gateway Node we can submit map reduce jobs using hadoop jar command. https://gist.github.com/dgadiraju/0d3df07693e78d07164af0c14493707d There will be JVM launched on the gateway node. It will talk to Resource Manager and get …
Now let us talk about YARN Application Life Cycle. YARN is the resource management framework. We can use distributed data processing frameworks such as Map Reduce, Spark etc., by plugging into YARN. A YARN application can be Map Reduce Job or Spark Application. From YARN perspective data is being processed by containers. Let us understand …
Both YARN and MR2 have command line interface. YARN CLI can be used to submit YARN applications and manage them Map Reduce CLI can be used to submit jobs using job files and also to manage them There are hand full of commands which Hadoop Administrator should be aware of. Type yarn or mapred and …
Now let us review the web interfaces and also log files to troubleshoot any YARN or Map Reduce specific issues. Web UIs Resource Manager Web UI Job History Server Web UI Using Cloudera Manager to troubleshoot the issues of Resource Manager components. Using Resource Manager Web UI to troubleshoot the jobs Resource Manager Web UI …
As we have changed the properties with respect to node manager capacity, let us run randomtextwriter again and see how long it take. We can override individual properties at runtime using -D and multiple properties using -conf and xml file similar to yarn-site.xml or mapred-site.xml. https://gist.github.com/dgadiraju/f2852840916b1e79f4fb6830d93c8b22 Now let us run word count program from hadoop examples …
configuring files and important properties – Running Jobs Read More »
