Let us understand different licensing options as well as installation options. Different licensing option – Express vs. Enterprise license We are not going to cover Enterprise licensing in detail. It is typically done between management of your enterprise and Cloudera sales team. Installation using Parcels Recommended by Cloudera We can create local repositories for parcels …
Cloudera Manager is the management tool for setup and manage clusters. It provides wizard to setup cluster as well as to configure alerts in case of any issues. Install Cloudera Manager – sudo yum -y install cloudera-manager-server Stop Cloudera Manager – sudo systemctl stop cloudera-scm-server Setup scm database – sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -h localhost scm root …
In this topic, we will set up data sets on our local machine for basic I/O operations By using git we clone the data sets into our local machine use this command in the terminal for setting up data sets git clone https://github.com/dgadiraju/data.git In labs, the data sets are pre-exist
Let us setup pre-requisites like JDK and MySQL database so that we can configure cloudera manager and other CDH components using external database. Install Java SDK that come with Cloudera – sudo yum install oracle-j2sdk1.7 -y Install mysql (mariadb) https://gist.github.com/dgadiraju/c35eb63a77438b5bacacdd43ca14ba74 Setup mysql connector – sudo yum -y install mysql-connector-java We need to install mysql-connector-java on other nodes …
Now let us look into the basic map reduce operations such as Map, Filter, Reduce and Rich Collection API. Problem Statement Get Sum of squares all Even Numbers in a given range val l = (1 to 100).toList val f = l.filter(ele => ele % 2 == 0) val m = f.map(rec => rec * …
As part of this topic we will see one of the important topics for building Spark applications using Scala i.e Collections. Collections Scala collections are categorized into 3 types Seq Set Map Seq Sequence have length Elements in Sequence can be accessed using prefix eg: scala.Array, scala.collection.immutable.List etc Classes are divided into Seq classes and Buffer …
