Let us see how to setup Hive on an existing cluster using Cloudera Manager. Configure Hive Go to Add Service and select Hive Assign role or select the host to install the components – Gateway, HiveServer2 and Hive Metastore Server. We will be configuring Gateway on all nodes and Hive Server as well as Hive …
As part of this section, we will see how to set up the Hive in the Cloudera distribution. We will also understand important concepts related to Hive. Setup Hive and Impala Validating Hive and Impala Components and Properties of Hive Troubleshooting Hive issues Hive Commands and Queries – Overview Different Query Engines Components and Properties …
Spark provides APIs as well as Framework for distributed processing. Even though there are differences with respect to the development of applications using Spark 1.6.x versus Spark 2.3.x, deployment does not change much. Instead of spark-submit, we need to use spark2-submit. Same is the case with spark-shell and pyspark, we have to use spark2-shell and …
Let us review some of the important concepts in Google Cloud Platform. Instance Templates Instance Groups VM Instances Boot Disks Additional Storage or Disks Internal IP and External IP SSH Options – gcloud and regular SSH Instance states – Start, Stop, Reset, Delete Firewall Rules
Now let us provision server for our Big Data Cluster. Go to Instance Templates Click on more options and then choose Create VM Name: bigdataserver-1 Click on Create Once the server is provisioned, we need to add additional disks for storage. Click on bigdataserver-1 Click on Edit Click on Add Item under Additional disks Click on …
Once parcels is setup we can setup Spark 2.3.x on the existing cluster. Download Oracle JDK 1.8 on all the servers – ansible all -i hosts -a ” wget –no-check-certificate -c –header ‘Cookie: oraclelicense=accept-securebackup-cookie’ http://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.rpm ” –private-key=~/.ssh/google_compute_engine Install Oracle JDK 1.8 on all the servers – ansible all -i hosts -a ” rpm -ivh jdk-8u191-linux-x64.rpm ” –become –private-key=~/.ssh/google_compute_engine …
