Let us see different components related to Namenode High Availability. You can refer to this nicely written article on Namenode HA as part of Hadoop 2.x. Zookeeper – We already have Zookeeper running on our cluster. It will be used as part of Namenode High Availability. Quorum Based Storage – Active Namenode writes to it and …
Now let us understand key concepts with respect to HDFS Namenode High Availability. Let us recap HDFS components before we get into HA. HDFS have Namenode, Secondary Namenode and Datanodes When data is saved in the form of files in HDFS Files will be divided into blocks and blocks are saved in Datanodes Mapping between …
HDFS Namenode HA – Quick Recap of HDFS typical Configuration Read More »
Now let us review core-site.xml and hdfs-site.xml properties to understand how configurations are defined. In core-site.xml, fs.defaultFS will be pointing to nameservice1 instead of static IP address and port number (e.g.: hdfs://nameservice1) In hdfs-site.xml we will see new property by name dfs.nameservices and value nameservice1 For both Namenodes, we will have logical name defined under …
Let us see how we can configure High Availability for HDFS Namenode. It can be done using Shared Edits or Journal Nodes. However, using Cloudera Manager only Journal Nodes is supported. Prerequisite – Make sure zookeeper is installed and running along with HDFS components such as Namenode, Secondary Namenode, Datanodes. Let us review what are …
Let us see high level overview about High Availability for HDFS as well as YARN. Namenode is master for HDFS and Resource Manager is master for YARN. Both Namenode, as well as Resource Manager, are a single point of failure. If Resource Manager is down, no data processing can be done in the cluster. None …
