Even though we can configure Spark 2.3.x on Cloudera based cluster using Cloudera Manager, we need to use Parcels. Cloudera recommends parcels over packages to build the cluster. We do not need to set up local repositories with parcels. Cloudera Manager caches the parcels repositories on the node where it is running and takes care …
Let us create template for Big Data Server. We will be using this template to provision servers to build Big Data Cluster. First check quotas for CPUs, use 2 vCPUs with 8 GB. If you want you can use higher configuration if the quota is higher than 24 CPUs in any one region. You also need …
Let us understand the Execution Life Cycle of Spark. You can review this using Spark Official Documentation. We submit the job for the client. The JVM typically acts as the Driver Program. It will talk to the Resource Manager and create the Application Master. Application Master will talk to Worker Nodes on which Node Managers are …
Let us review some of the important properties of Spark. Like Map Reduce, Spark will create containers to process data. These containers are called as Executors When it comes to Map Reduce – Map Tasks are based on number of blocks of underlying file, where as with Spark it will create containers based up on …
Even though Cloudera support both Spark 1.6.x as well as Spark 2.3.x, we can only setup Spark 1.6.x with Packages. Go to the Cloudera Manager Dashboard Click on Add Service in drop down of the cluster Choose Spark 1.6.x (don’t choose Stand Alone) We will be using bigdataserver-4 as Spark Job History Server. Review properties …
Let us sign up to GCP and start using it for setting up the clusters. We will be demonstrating using GCP quite extensively. Home Page: https://cloud.google.com $300 credit for an year. This credit is enough to learn set up process for most of the clusters. We will be able to stop the servers and start …
