Instead of waiting until long-running to be close to complete, resources will be allocated to all the subsequent jobs in Fair Manner.
- Available resources will be shared evenly between all the outstanding jobs.
- By default, Cloudera Hadoop Distribution uses Fair Scheduler.
- Configuration files related to the fair scheduler
- yarn-site.xml
- fair-scheduler.xml – allocation file
- To customize the Fair Scheduler, set configuration properties in yarn-site.xml and update the Fair Scheduler allocation file to add new queues or update existing queues. We can change the Scheduling Policy, update allocations and assign weights to the queues.
- E.g.: We will be defining the queues as below.
- Defining 3 Primary queues in the form of XML file.
- Prod Queue with 80% weight – root.prod
- QA Queues with 10% Weight – root.qa
- ETL Queue with 10% weight – root.etl
- We can also specify scheduling policy within each queue, drf or fair or fifo
- The default is drf – Dominant Resource Fairness. While the fair is based on Memory, drf is based on multiple resource types.
- Defining 3 Primary queues in the form of XML file.
