Let us understand the execution modes as well as different components of the Spark Framework. Also, we will recap some important aspects of YARN. Execution Modes Following are the different execution modes supported by Spark. Local (for development) Standalone (for development) Mesos YARN As our cluster uses YARN, let us recap some important aspects of …
Let us start with a simple application to understand details related to architecture using pyspark. As we have multiple versions of Spark on our lab and we are exploring Spark 2 we need to export SPARK_MAIOR_VERSION with 2. For this demo, we will disable dynamic allocation by setting spark. dynamicAllocation.enabled to false. Launch pyspark using …
We need to deal with 2 types of data structures in Spark – RDD and Data Frames. We will see both in detail as we proceed further. RDD is there for quite some time and it is the low-level data structure which spark uses to distribute the data between tasks while data is being processed. …
In the earlier versions of Spark, we have core API at the bottom and all the higher level modules work with core API. Examples of core API are a map, reduce, join, groupByKey etc. But with Spark 2, Data Frames and Spark SQL has become the core module. Core – Transformations and Actions — APIs …
Let us have a quick review of Core APIs that are available in Spark. We will cover Data Frame APIs and Spark SQL at a later point in time. SparkContext exposes APIs such as textFile, sequenceFile to read data from files into a distributed collection called as RDD. RDD stands for Resilient Distributed Dataset and …
Let us understand how Spark Official Documentation is organized. Overview Start Documentation for Different Modules API Docs Cluster Overview under Deploying Submitting Applications Execution Frameworks Tuning and more
