So far, we have seen setting up Zookeeper and HDFS, now we will see how to setup YARN which stands for Yet Another Resource Negotiator. YARN (MRv2) introduces newer daemons that are responsible for job scheduling/monitoring and resource management. Setup YARN + MR2 Run Simple Map Reduce Job Components of YARN and MR2 Configuration Files …
There are several commands to perform administration using CLI. We need to use HDFS super user to manage HDFS cluster using commands. In our case it is hdfs itself. Formatting Namenode Rolling Edits Save Namespace (create fsimage) Enter or Leave Safemode Running balancer Running file system utility (fsck) and many more By this time you …
HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system. Some common use cases of snapshots are data backup, protection against user errors and disaster recovery. It does not copy actual data. It will keep track of changes to …
Now let us have a look at HDFS usage commands and also commands used to get the metadata. hadoop fs -df – to get details about the amount of storage used by HDFS. Use -s to get summarized information and -h to get information in readable format. hadoop fs -du – to get the size …
Let us see how we can override properties while running commands such as hadoop fs. Let us first review properties files such as core-site.xml and hdfs-site.xml. We can override any non-final property using -Dproperty_name=property_value as part of hadoop fs command. We can also use options such as -fs to override Namenode URI We can also change replication …
https://gist.github.com/dgadiraju/fed51f71eb1eb9a086eaefc03b3cd577
