Let us create template for Big Data Server. We will be using this template to provision servers to build Big Data Cluster. First check quotas for CPUs, use 2 vCPUs with 8 GB. If you want you can use higher configuration if the quota is higher than 24 CPUs in any one region. You also need …
Let us understand the Execution Life Cycle of Spark. You can review this using Spark Official Documentation. We submit the job for the client. The JVM typically acts as the Driver Program. It will talk to the Resource Manager and create the Application Master. Application Master will talk to Worker Nodes on which Node Managers are …
Let us review some of the important properties of Spark. Like Map Reduce, Spark will create containers to process data. These containers are called as Executors When it comes to Map Reduce – Map Tasks are based on number of blocks of underlying file, where as with Spark it will create containers based up on …
Even though Cloudera support both Spark 1.6.x as well as Spark 2.3.x, we can only setup Spark 1.6.x with Packages. Go to the Cloudera Manager Dashboard Click on Add Service in drop down of the cluster Choose Spark 1.6.x (don’t choose Stand Alone) We will be using bigdataserver-4 as Spark Job History Server. Review properties …
Let us sign up to GCP and start using it for setting up the clusters. We will be demonstrating using GCP quite extensively. Home Page: https://cloud.google.com $300 credit for an year. This credit is enough to learn set up process for most of the clusters. We will be able to stop the servers and start …
As part of this section, we will see how to set up Spark components while exploring some of the key concepts of this very important service. There are 2 versions of Spark, we will see how to setup both Spark 1.6 as well as Spark 2.3. However to setup Spark 2.3, we need to use …
