Now let us review core-site.xml and hdfs-site.xml properties to understand how configurations are defined. In core-site.xml, fs.defaultFS will be pointing to nameservice1 instead of static IP address and port number (e.g.: hdfs://nameservice1) In hdfs-site.xml we will see new property by name dfs.nameservices and value nameservice1 For both Namenodes, we will have logical name defined under …
Let us see how we can configure High Availability for HDFS Namenode. It can be done using Shared Edits or Journal Nodes. However, using Cloudera Manager only Journal Nodes is supported. Prerequisite – Make sure zookeeper is installed and running along with HDFS components such as Namenode, Secondary Namenode, Datanodes. Let us review what are …
Let us see high level overview about High Availability for HDFS as well as YARN. Namenode is master for HDFS and Resource Manager is master for YARN. Both Namenode, as well as Resource Manager, are a single point of failure. If Resource Manager is down, no data processing can be done in the cluster. None …
As part of this section we will see how to enable HDFS Namenode High Availability as well as YARN Resource Manager High Availability while exploring key concepts. High Availability – Overview Configure HDFS Namenode HA Review Properties – HDFS Namenode HA HDFS Namenode HA – Key Concepts Configure YARN Resource Manager HA Review – YARN …
Now let us talk about Map Reduce Job Execution Life Cycle. While YARN is Resource Management framework, Map Reduce is distributed data processing framework. On Gateway Node we can submit map reduce jobs using hadoop jar command. https://gist.github.com/dgadiraju/0d3df07693e78d07164af0c14493707d There will be JVM launched on the gateway node. It will talk to Resource Manager and get …
Now let us talk about YARN Application Life Cycle. YARN is the resource management framework. We can use distributed data processing frameworks such as Map Reduce, Spark etc., by plugging into YARN. A YARN application can be Map Reduce Job or Spark Application. From YARN perspective data is being processed by containers. Let us understand …
