Review Web UIs and log files
There are several Web UIs you need to be aware of with respect to HDFS. Namenode Web UI Datanode Web UI Using Cloudera Manager to troubleshoot issues Log files related to the service
Configuration files and Important Properties
Unlike plain vanilla distribution and other vendor distributions, Cloudera manages configuration files a bit different. Typically configuration files will be in /etc/hadoop/conf. But when it comes to Cloudera, /etc/hadoop/conf will only have templates. Actual properties files are managed under /var/run/cloudera-scm-agent/process on each node. hadoop-env.sh is for environment variables HADOOP_HOME JAVA_HOME HADOOP_HEAPSIZE – default heap size …
Components of HDFS contd
Anatomy of File Write in HDFS Anatomy of File Read in HDFS Features of HDFS Fault tolerant – HDFS uses mirroring at block level and dfs.replication controls how many copies should be made. Traditionally we use RAID for fault tolerance of Hard Drive failures in Network Storage. In Hadoop replication takes care of not only …
Components of HDFS
Now let us explore different components related to HDFS and how they are used to store files in HDFS. We need to configure the Namenode, Secondary Namenode, Datanodes and balancer etc as part of HDFS. Here Namenode act as master and Secondary Namenode as helper for Namenode where as Datanodes act as slaves. Actual data …
Copy Data into HDFS contd
A file will be divided into blocks (by default 128 MB) and those blocks will be physically stored as part of servers where datanode process is running. For example, a 1 GB file will be divided into 8 blocks of 128 MB each and a 200 MB file will be divided into 2 blocks of …
Copy Data into HDFS
Let us copy data into HDFS and understand details about how data is stored in HDFS. hadoop fs is the main command using which we can copy and manage files in HDFS. https://gist.github.com/dgadiraju/3ef7a526ff1b047f5955ff61bfd9c38c