Now let us get into the details about Namenode Recovery Process. Namenode recovery happens in safe mode. Automated Process Let us assume that namenode is restarted (we can do it using Cloudera Manager web interface) It will start in safe mode. In safe mode, we will not be able to add files to HDFS, but …
Process of creating new fsimage at regular intervals is known as checkpointing. A new fsimage is created on namenode when it is formatted and added. Changes to metadata will be logged in edit logs on Namenode. Secondary Namenode takes care of checkpointing. It will take the last fsimage (from Namenode or Secondary Namenode) and latest …
At times Namenode might be down due to planned maintenance or unplanned outages. As administrators it is very important for us to understand recovery process of Namenode. Let us start with Namenode metadata (information about files and blocks). There is metadata related to blocks as well as files. At block level we will have mapping …
There are several Web UIs you need to be aware of with respect to HDFS. Namenode Web UI Datanode Web UI Using Cloudera Manager to troubleshoot issues Log files related to the service
Unlike plain vanilla distribution and other vendor distributions, Cloudera manages configuration files a bit different. Typically configuration files will be in /etc/hadoop/conf. But when it comes to Cloudera, /etc/hadoop/conf will only have templates. Actual properties files are managed under /var/run/cloudera-scm-agent/process on each node. hadoop-env.sh is for environment variables HADOOP_HOME JAVA_HOME HADOOP_HEAPSIZE – default heap size …
Anatomy of File Write in HDFS Anatomy of File Read in HDFS Features of HDFS Fault tolerant – HDFS uses mirroring at block level and dfs.replication controls how many copies should be made. Traditionally we use RAID for fault tolerance of Hard Drive failures in Network Storage. In Hadoop replication takes care of not only …
