Introduction to Fair Scheduler

Instead of waiting until long-running to be close to complete, resources will be allocated to all the subsequent jobs in Fair Manner.

  • Available resources will be shared evenly between all the outstanding jobs.
  • By default, Cloudera Hadoop Distribution uses Fair Scheduler.
  • Configuration files related to the fair scheduler
    • yarn-site.xml
    • fair-scheduler.xml – allocation file
  • To customize the Fair Scheduler, set configuration properties in yarn-site.xml and update the Fair Scheduler allocation file to add new queues or update existing queues. We can change the Scheduling Policy, update allocations and assign weights to the queues.
  • E.g.: We will be defining the queues as below.
    • Defining 3 Primary queues in the form of XML file.
      • Prod Queue with 80% weight –
      • QA Queues with 10% Weight –
      • ETL Queue with 10% weight – root.etl
    • We can also specify scheduling policy within each queue, drf or fair or fifo
    • The default is drf – Dominant Resource Fairness. While the fair is based on Memory, drf is based on multiple resource types.

Share this post