As part of this section, we will see how to set up the Hive in the Cloudera distribution. We will also understand important concepts related to Hive. Setup Hive and Impala Validating Hive and Impala Components and Properties of Hive Troubleshooting Hive issues Hive Commands and Queries – Overview Different Query Engines Components and Properties …
Spark provides APIs as well as Framework for distributed processing. Even though there are differences with respect to the development of applications using Spark 1.6.x versus Spark 2.3.x, deployment does not change much. Instead of spark-submit, we need to use spark2-submit. Same is the case with spark-shell and pyspark, we have to use spark2-shell and …
Let us review some of the important concepts in Google Cloud Platform. Instance Templates Instance Groups VM Instances Boot Disks Additional Storage or Disks Internal IP and External IP SSH Options – gcloud and regular SSH Instance states – Start, Stop, Reset, Delete Firewall Rules
Now let us provision server for our Big Data Cluster. Go to Instance Templates Click on more options and then choose Create VM Name: bigdataserver-1 Click on Create Once the server is provisioned, we need to add additional disks for storage. Click on bigdataserver-1 Click on Edit Click on Add Item under Additional disks Click on …
Once parcels is setup we can setup Spark 2.3.x on the existing cluster. Download Oracle JDK 1.8 on all the servers – ansible all -i hosts -a ” wget –no-check-certificate -c –header ‘Cookie: oraclelicense=accept-securebackup-cookie’ http://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.rpm ” –private-key=~/.ssh/google_compute_engine Install Oracle JDK 1.8 on all the servers – ansible all -i hosts -a ” rpm -ivh jdk-8u191-linux-x64.rpm ” –become –private-key=~/.ssh/google_compute_engine …
Even though we can configure Spark 2.3.x on Cloudera based cluster using Cloudera Manager, we need to use Parcels. Cloudera recommends parcels over packages to build the cluster. We do not need to set up local repositories with parcels. Cloudera Manager caches the parcels repositories on the node where it is running and takes care …
