Day: February 10, 2023

https://gist.github.com/dgadiraju/0ff21e302882e6565ca02aeb068fd287

https://gist.github.com/dgadiraju/e8b8687e00a8b43087348068e536c18d

Unlike plain vanilla distribution and other vendor distributions, Cloudera manages configuration files a bit different. Typically configuration files will be in /etc/hadoop/conf. But when it comes to Cloudera, /etc/hadoop/conf will only have templates. Actual properties files are managed under /var/run/cloudera-scm-agent/process on each node. hadoop-env.sh – for memory settings of Resource Manager, Node Manager etc. core-site.xml …

Configuration files and Important Properties – Overview Read More »

Now let us explore different components related to YARN as well as Map Reduce 2 and how they are used in Resource Management as well as processing data. YARN stands for Yet Another Resource Negotiator. It provides capabilities related to Resource Management and actual data processing is done by frameworks such as Map Reduce, Spark, …

Components of YARN and MR2 Read More »

Let us run simple Map Reduce Job and see what happens. We will be using Hadoop examples that come as part of the setup process itself. We can use Hadoop jar or yarn jar to submit map reduce job as YARN application. Let us run an application called randomtextwriter which will generate 10 GB of …

Run Simple Map Reduce Job Read More »

Here are the steps involved in setting up YARN + MR2 using Cloudera Manager. There are 2 different processing engines that can be configured using Cloudera Manager (Map Reduce which is a legacy framework and YARN + MR2). We don’t need to worry much about the legacy framework. Choose the drop-down of cluster Cluster 1 …

Setup YARN + MR2 Read More »