Let us understand the execution modes as well as different components of the Spark Framework. Also, we will recap some important aspects of YARN.
Execution Modes
Following are the different execution modes supported by Spark.
- Local (for development)
- Standalone (for development)
- Mesos
- YARN
As our cluster uses YARN, let us recap some important aspects of YARN.
- YARN uses Master (Resource Manager) and Slave (Node Managers) Architecture.
- YARN primarily takes care of resource management and scheduling the tasks.
- For each YARN Application, there will be an application master and set of containers created to process the data.
- We can plugin different distributed frameworks into YARN, such as Map Reduce, Spark etc.
- Spark creates executors to process the data and these executors will be managed the Resource Manager and per job Application Master.
Execution Framework
Let us understand the Spark Execution Framework by running wordcount program using ROD.
- Driver Program
- Spark Context
- Executors
- Executor Cache
- Executor Tasks
- Job
- Stage
- Task (Executor Tasks)