As part of this topic, we will explore Basic I/O operations Read data from files Convert into collection Perform collection operation to preview the data Run map-reduce operations Reading data from files To read files from a file system in scala there is a package called scala.io In scala.io we have a package named Source which …
Basic I/O Operations and using Scala Collections APIs Read More »
As highlighted earlier replication will cover all system failures and redundant networking will cover cable and network card failures. But it will not cover the rack maintenance or network switch failures. Rack is nothing but container where we place multiple servers in it. Also replication can only cover n-1 nodes in a m node cluster …
Now let us get into the details about Namenode Recovery Process. Namenode recovery happens in safe mode. Automated Process Let us assume that namenode is restarted (we can do it using Cloudera Manager web interface) It will start in safe mode. In safe mode, we will not be able to add files to HDFS, but …
Process of creating new fsimage at regular intervals is known as checkpointing. A new fsimage is created on namenode when it is formatted and added. Changes to metadata will be logged in edit logs on Namenode. Secondary Namenode takes care of checkpointing. It will take the last fsimage (from Namenode or Secondary Namenode) and latest …
At times Namenode might be down due to planned maintenance or unplanned outages. As administrators it is very important for us to understand recovery process of Namenode. Let us start with Namenode metadata (information about files and blocks). There is metadata related to blocks as well as files. At block level we will have mapping …
There are several Web UIs you need to be aware of with respect to HDFS. Namenode Web UI Datanode Web UI Using Cloudera Manager to troubleshoot issues Log files related to the service
